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Abstract 


A  Survey  on  Interoperability  Measurement 

For  nearly  thirty  years,  both  government  and  industry  have  actively  explored  research  on 
interoperability  measurement  with  the  goal  of  creating  a  straightforward  way  of  measuring, 
reporting,  then  improving  the  interoperability  of  complex  networks  of  people,  equipment, 
processes  and  organizations.  Researchers  have  created  frameworks  and  models,  proposed 
measures,  described  levels,  and  listed  a  variety  of  qualitative  factors  in  support  of  an 
interoperability  measure.  Within  extant  interoperability  research,  the  authors’  research  has 
uncovered  nearly  three  dozen  definitions  of  interoperability,  over  five  dozen  distinct  types  of 
interoperability,  numerous  interoperability  attributes,  and  fourteen  foundational  interoperability 
measurement  models  and  methodologies.  At  least  eleven  research  groups  have  been  the  centers- 
of-gravity  for  interoperability  measurement  research.  This  survey  paper  summarizes  and  focuses 
the  current  body  of  knowledge  on  interoperability  measurement  and  identifies  areas  where  further 
research  is  needed. 


Background 

Interoperability  has  been  highlighted  as  a  problem  within  the  Department  of  Defense 
(DoD)  since  1965  when  the  Special  Subcommittee  on  Tactical  Air  Support  of  the  House  Armed 
Services  Committee  stated  that  with  regards  to  close  air  support,  the  “incompatibility  of  Army 
and  Air  Force  radios  was  “appalling”  and  they  called  the  situation  a  “communications  fiasco.” 
[GAO,  1987,  p.  20]  The  DoD  responded  in  1967  by  publishing  an  interoperability  directive,  DoD 
Directive  4630.5  which  “established  policy  and  procedures  to  ensure  that  C3  equipment  [could] 
interoperate.”  [Ibid]  But  after  continued  Congressional  criticism  regarding  the  directive  and  its 
implementation  throughout  the  1970s,  “the  Secretary  of  Defense  informed  the  Congress  [in  1977] 
that  the  directive  would  be  revised.”  [Ibid]  By  1985  “the  directive  had  still  not  been  revised” 
prompting  the  Senate  Armed  Services  Committee  to  warn  the  Secretary  of  Defense  that  they 
might  “consider  a  legislative  restriction  on  the  expenditure  of  any  funds  for  communications 
equipment  unless  meaningful  progress  is  made.”  [Ibid]  The  Directive  was  revised  and  published 
that  same  year. 

Although  the  basic  policies  were  now  established,  in  April  1987  the  GAO  responded  to  a 
request  by  the  Chairman  of  the  Legislation  and  National  Security  Subcommittee  within  the  House 
of  Representatives  asking  the  GAO  to  describe  “the  extent  to  which  communications 
interoperability  problems  have  been  identified  during  exercises  and  past  operations”  and  “the 
impediments  preventing  interoperability  and  how  much  they  can  be  overcome”  [GAO,  1987,  p.  1] 
The  GAO  responded  that  interoperability  has  been  a  “longstanding  problem,”  and  that  the 
“services  historically  have  been  unable  to  communicate  effectively  among  themselves  during 
joint  operations  and  exercises.”  [GAO,  1987,  p.  8]  The  GAO  stated  examples  of  interoperability 
failures  in  operations  in  “Korea,  the  Dominican  Republic  Landing,  Vietnam,  and. .  .the  Grenada 
intervention  in  1983.”  [Ibid]  Half  a  decade  later  in  response  to  another  Congressional  request,  the 
GAO  acknowledged  that  the  DoD  “has  worked  hard  over  the  years  to  achieve  greater 
interoperability,”  but  it  continued  to  experience  interoperability  problems  during. .  .the  Persian 
Gulf  War  in  1991.”  [GAO,  1993,  p.  2]  In  1992,  the  Chairman  of  the  Joint  Chiefs  of  Staff 
“announced  a  new  initiative  called  ‘Joint  C4I  for  the  Warrior,”’  which  was  created  to  provide  a 
“common  global  vision”  for  the  services.  [Ibid]  Major  results  of  this  initiative  were  the  Defense 
Information  Services  Agency  (DISA)  interoperability  certification  process  in  1992,  the  C4ISR 


Architecture  Framework  published  in  1 996,  the  Defense  Information  Infrastructure  (DII)  Master 
Plan  published  in  1 994,  and  the  1 993  Levels  of  Information  Systems  Interoperability  (LISI) 
initiative  which  was  completed  in  1998.  [GAO,  1998,  pp.  20-21]  Great  progress  on  the 
interoperability  front  had  been  made  and  now  that  policies,  procedures,  organizations,  and 
oversight  mechanisms  were  in  place,  it  was  time  to  refine  interoperability  improvement.  The  way 
ahead  was  interoperability  measurement — a  trail  partially  blazed  by  LISI.  As  Kasunic  and 
Anderson  put  it,  “management  must  be  able  to  measure  what  they  wish  to  change.”  [Kasunic  & 
Anderson,  2004,  p.  16]  Further,  DoD  Instruction  4630.8,  Procedures  for  Interoperability  and 
Supportability  of  Information  Technology  (IT)  and  National  Security  Systems  (NSS),  requires 
interoperability  assessment  through  “measurable,  performance-based  criteria.”  [DoDI  4630.8, 
2004,  p.  29] 

To  this  end,  this  paper  describes  a  survey  of  research  on  interoperability  measurement 
spanning  nearly  three  decades.  This  paper  not  only  summarizes  the  extant  research,  but  identifies 
gaps  and  areas  for  further  research. 


Introduction 

Although  a  comprehensive  survey  on  a  topic  as  broad  as  interoperability  is  impossible, 
our  survey  was  extensive  and  we  limited  it  to  the  sub-topic  of  interoperability  measurement.  We 
occasionally  reference  a  paper  which,  while  not  focused  on  interoperability  measurement, 
indirectly  supported  the  topic.  Also,  although  we  did  not  constrain  our  literature  search  to  DoD- 
focused  interoperability  measurement,  we  discovered  that  much  has  been  published  either  in 
support  of  the  DoD  or  by  the  DoD  and  its  associated  organizations.  We’ve  organized  our 
research  in  the  following  manner.  To  set  the  stage,  we  first  give  an  analysis  and  history  of 
interoperability  definitions  from  the  past  thirty  years  and  then  present  a  short,  but  comprehensive 
survey  of  all  the  different  types  of  interoperability  mentioned  in  research  work  over  the  same  time 
period.  With  definitions  and  types  of  interoperability  as  the  foundation,  we  then  name  what  we 
call  interoperability  measurement  “centers-of-gravity”  which  are  the  organizations  which  have 
produced  a  method  for  measuring  interoperability.  We  then  describe  and  provide  a  critique  of 
their  fourteen  major  interoperability  measurement  models,  methodologies,  and  processes.  We 
end  this  research  paper  with  a  discussion  of  the  mathematics  of  interoperability  measurement,  the 
lack  of  institutionalization  of  twelve  of  the  fourteen  methods,  and  identification  of  and 
recommendation  for  filling  the  current  gaps  in  interoperability  measurement  research. 

Interoperability  Definitions 

As  expected,  there  have  been  numerous  definitions  of  interoperability  put  forth  by 
researchers,  standards  bodies,  and  the  government  over  the  past  thirty  years.  While  a 
comprehensive  listing  would  be  unproductive,  we  have  identified  thirty-four  distinct  definitions 
of  interoperability  used  in  research  papers,  standards,  and  other  government  documents  during  the 
time  period.  These  definitions  are  listed  in  the  appendix  in  chronological  order.  Next  to  each 
definition  is  a  list  of  sources  which  either  introduce  or  reference  the  definition.  Although  most  of 
the  definitions  pertain  to  interoperability  in  general,  eight  define  specific  types  of  interoperability 
(e.g.,  technical  interoperability,  logistic  interoperability,  and  operational  interoperability).  The 
definitions  pertaining  to  a  specific  type  of  interoperability  are  clearly  annotated  in  the  table. 
Finally,  the  origin  of  the  definition  are  marked  as  either  DoD,  Standard,  or  Other  where  DoD 
refers  to  any  DoD  or  DoD-related/contracted  organization,  Standard  refers  to  any  standard 
making  body  such  as  the  Institute  of  Electrical  and  Electronics  Engineers  (IEEE),  and  Other 
refers  to  all  other  sources. 


Besides  the  number  of  interoperability  definitions,  it  is  interesting  to  note  the  obvious 
similarities  and  differences  between  many  of  the  definitions.  To  begin  our  analysis  of  these 
interoperability  definitions,  we  begin  by  presenting  a  histogram  in  Figure  1  which  shows  how 
many  different  interoperability  definitions  were  proposed  in  research  papers,  reports,  standards, 
or  government  documents  each  year  since  1977.  This  histogram  gives  us  a  feel  for  where  in  time 
interoperability  became  a  focus  not  only  within  the  DoD,  but  without  as  well. 


Figure  1  Number  of  Interoperability  Definitions  per  Year 

An  apparent  surge  in  interest  in  interoperability  is  evident  beginning  in  1 995  and  ending 
in  2003.  This  is  corroborated  by  the  vignette  on  the  history  of  interoperability  presented  in  the 
background  section  of  this  paper.  As  we  will  show  later  in  this  paper,  1995-2003  is  the  time 
period  in  which  most  research  on  interoperability  measurement  also  occurred.  The  implication  is 
obvious — before  creating  a  model  for  interoperability  measurement,  one  had  to  first  understand 
what  interoperability  was.  Much  of  the  remainder  of  this  paper  will  focus  on  this  time  period. 

The  histogram  in  Figure  1  also  shows  that  there  has  been  an  apparent  decrease  in  the 
number  of  interoperability  definitions  put  forward  in  research,  reports,  standards,  and  government 
documents  since  2004.  We  believe  this  decline  signals  the  beginning  of  a  time  period  in  which 
one  or  more  definitions  have  become  accepted  by  a  wide  variety  of  researchers,  standards 
organizations,  and  the  government.  Figure  2  substantiates  this  by  showing  the  popularity  of  each 
definition — in  other  words,  the  number  of  research  papers,  reports,  standards,  or  government 
documents  using  the  definition.  Definition  #1,  by  far,  is  referenced  the  most. 


Definition  # 


Figure  2  Interoperability  Definition  Popularity 


Definition  #1  was  the  oldest  definition  of  interoperability  we  found  in  our  literature 
search,  and  it  is  still  commonly  used  in  DoD  documents  today.  This  1977  definition  of 
interoperability  was  likely  in  use  within  the  DoD  as  far  back  as  1 967  when  the  first  publication  of 
DoD  Directive  4630.5  was  made.  A  close  reading  of  all  the  interoperability  definitions  presented 
in  this  paper  indicates  that  many  of  them,  especially  the  DoD-related  definitions,  represent  re¬ 
wordings  or  subsets  of  the  1977  definition.  Definition  #1  is  repeated  below  for  convenience. 

INTEROPERABILITY :  The  ability  of  systems,  units,  or  forces  to  provide 
services  to  and  accept  services  from  other  systems,  units,  or  forces  and  to  use  the 
services  so  exchanged  to  enable  them  to  operate  effectively  together. 

Interoperability  Types 

We  believe  one  reason  this  definition  of  interoperability  has  been  well  accepted  over  the 
past  thirty  years  is  that  it  is  flexible  enough  to  cover  many  different  types  of  interoperability.  Its 
wording  talks  about  systems,  possibly  implying  technical  types  of  interoperability,  yet  also 
mentions  units  and  forces,  implying  operational  types  of  interoperability.  Our  research  supports 
the  idea  that  most  (if  not  all)  types  of  interoperability  can  be  classified  as  either  technical  or  non¬ 
technical.  Examples  of  technical  interoperability  types  are  communications,  electronic, 
application,  and  multi-database  interoperability.  Examples  of  non-technical  interoperability  types 
include  organizational,  operational,  process,  cultural,  and  coalition  interoperability.  Although  we 
have  stated  these  examples  of  technical  and  non-technical  interoperability,  note  that  the 
classification  of  an  interoperability  type  as  technical  versus  non-technical  is  subject  to 
interpretation.  In  fact  the  “binning”  of  interoperability  types  may  be  situation  dependent.  For 
example,  Public  Service  interoperability  may  be  considered  non-technical  from  an  enterprise 
point  of  view,  but  would  be  categorized  as  technical  when  considering  water  pipe  capacities, 
electric  line  loads,  or  telephone  switches.  Another  reason  definition  #1  is  so  versatile  is  that  it 
talks  in  generalities  about  services  rather  than  specifics  such  as  packets,  messages,  orders, 
deliveries,  or  procedures.  Finally,  it  simply  makes  the  point  that  the  exchange  of  services  creates 
an  effective  operation  of  the  systems,  units,  or  forces. 

Our  research  produced  a  list  of  sixty-four  different  types  of  interoperability  mentioned  in 
research  papers,  reports,  standards,  and  government  documents  over  the  past  thirty  years.  Very 
few  of  these  types  were  specifically  defined  in  their  source  document,  but  all  demonstrate  the 
richness  of  the  interoperability  field  of  study.  These  interoperability  types  are  listed,  along  with 
their  source  in  the  appendix  in  chronological  order  of  their  appearance. 

Interoperability  Measurement  Research  Centers  of  Gravity 

Over  the  past  thirty  years  there  have  been  numerous  research  papers,  magazine  articles, 
and  presentations  published  and  conferences  held  on  interoperability-related  topics.  Many  of 
these  have  only  marginally  extended  the  body  of  knowledge,  adding  just  a  change  in  context  or 
capitalized  on  interoperability  as  a  “buzzword.”  Substantially  fewer  papers,  articles, 
presentations,  and  conferences  have  focused  on  the  topic  of  interoperability  measurement. 
Through  our  literature  search,  we  have  determined  that  there  are  eleven  organizations  which  have 
contributed  substantially  to  the  area  of  interoperability  measurement  by  publishing  one  or  more 
interoperability  measurement  models,  methodologies,  or  processes  (hereafter  referred  to  as 
models).  These  organizations  are  listed  in  Table  1  in  chronological  order  of  when  they  published 
their  first  interoperability  measurement  model.  Of  the  eleven  organizations  listed,  ten  are  either 
part  of,  or  directly  support  the  Departments  of  Defense  of  the  United  States,  Australia,  and 
Poland.  The  one  non-DoD-related  organization  (VMASC)  is  a  center  within  a  public  university 


which  has  military  modeling  and  simulation  as  one  of  its  six  core  capabilities  [VMASC  ‘07].  Of 
the  ten  DoD-related  centers-of-gravity,  two  (MITRE  Corp.  and  CMU-SEI)  are  Federally  Funded 
Research  and  Development  Centers  (FFRDC).  Four  (VMASC,  CMU-SEI,  Military  University  of 
Technology,  and  AFIT)  centers-of-gravity  are  either  educational  institutions  or  part  of 
educational  institutions. 


Table  1  Interoperability  Measurement  Research  Centers-of-Gravity 


Organization 

Model 

Defense  Information  Systems  Agency  (DISA) 

SoIM  (’80) 

MITRE  Corporation 

QoIM  (’89) 

LISI  (’98) 

Military  University  of  Technology,  Warsaw,  Poland 

MCISI  (’96) 

Joint  Theater  Air  and  Missile  Defense  Organization  (JTAMDO)  Contractor  SIM,  Inc. 

I AM  (’98) 

Australian  Defence  Science  and  Technology  Organisation  (DSTO) 

OIM  (’99) 

OIAM  (’05) 

Joint  Forces  Command  (JFCOM)  Joint  Forces  Program  Office  (JFPO) 

Stoplight  (’02) 

Old  Dominion  University  Virginia  Modeling  Analysis  and  Simulation  Center  (VMASC) 

LCI  (’03) 

LCIM  (’03) 

North  Atlantic  Treaty  Organization  (NATO) 

NMI(‘03) 

Carnegie  Mellon  Software  Engineering  Institute  (CMU-SEI) 

SoSI  (’04) 

Defence  Science  and  Technology  Laboratory  (Dstl)  and  Contractor,  QinetiQ,  pic 

NTI  (’04) 

Air  Force  Institute  of  Technology  (AFIT) 

i-Score  (’07) 

Interoperability  Measurement  Models 

Although  many  papers,  articles,  and  presentations  would  have  us  believe  that  LISI  is  the 
only  interoperability  measurement  model  available,  there  are  actually  at  least  fourteen  models 
available  for  use.  Each  one  of  the  models  has  a  specific  focus  and  some  are  more  capable  than 
others.  Some  have  been  widely  accepted,  and  others  are  rarely  used.  There  have  been  technical 
reports  published  which  summarize  some,  but  not  all,  of  the  interoperability  measurement  models 
listed  in  Table  1.  This  section  of  the  paper  seeks  to  describe  all  fourteen  models,  point  out  the 
strengths  and  uses  of  each,  and  reference  the  reader  to  other  critiques  or  reviews  of  these  models, 
including  their  original  source  documents.  We  frame  our  analysis  of  these  fourteen  models  with  a 
timeline. 


Spectrum  of  Interoperability  Model  (SoIM) 


Gilbert  LaVean,  an  employee  of  a  predecessor  organization  to  the  current  Defense 
Information  Systems  Agency  (D1SA),  acknowledged  in  1980,  in  the  IEEE  Transactions  on 
Communications,  that  inter-system  interoperability  was  poor  because  there  existed  a  “lack  of  a 
measure  of  interoperability  by  which  to  state  goals  for  specific  systems.”  [Gilbert,  1980,  p.  1449] 
In  order  to  combat  this  deficiency,  he  created  a  spectrum  of  interoperability  model  (which  we 
nickname  SoIM).  [Ibid,  p.  1448]  He  began  his  model  by  defining  the  two  most  important 
measures  of  interoperability  (technical  possibility  and  management/control  possibility).  [Ibid] 
The  technical  measure  was  a  number  between  1  (impractical  to  interface)  and  4  (common 
equipment  used)  and  the  management/control  measure  was  a  number  from  1  (complete 
independence  between  systems)  and  6  (separate  systems  placed  under  common 
management/control,  thus  becoming  the  same  system).  [Ibid]  He  then  stated  that  by  “combining 
these  two  measures,  it  is  possible  to  derive  a  spectrum  of  interoperability  that  permits  cost-versus- 
benefits  tradeoffs.”  His  combination  of  these  two  measures  resulted  in  seven  levels  of 
interoperability  which  are  listed  in  Table  2.  [Ibid]  Mr.  LaVean  recognizes  that  the  level  of 
interoperability  may  be  different  for  each  service  that  pairs  of  systems  provide  to  each  other,  so 
he  proposes  a  visualization  method,  which  he  calls  an  interoperability  matrix,  which  lists  services 
on  the  rows  of  the  matrix  and  levels  of  interoperability  on  the  columns.  [Ibid]  He  further 
proposes  a  current  view  and  a  “future”  view  of  the  interoperability  matrix  in  order  to  show 
evolution  of  the  systems  over  time.  [Ibid]  Thus,  the  purpose  of  the  LaVean  Spectrum  of 
Interoperability  Model  was  to  provide  a  simple  tool  for  program  managers  to  assess  current 
interoperability  of  their  systems  and  services,  to  set  goals  for  future  interoperability,  and  to 
visualize  the  current  and  future  states  of  interoperability.  Although  Mr.  Gilbert’s  Spectrum  of 
Interoperability  Model  was  groundbreaking  and  is  the  earliest  model  for  measuring 
interoperability  that  we  have  discovered,  we  were  able  to  find  no  further  mention  of  his  model 
after  its  original  publication  and  whether  or  not  it  was  used  by  program  managers  to  improve 
inter-system  interoperability  is  unknown.  Our  literature  search  did  not  reveal  that  Mr.  LaVean’s 
model  was  further  refined  by  other  researchers  after  its  original  publication. 


Table  2  SoIM  Levels  of  Interoperability 


Level  # 

Name 

Technical 

Measure 

Management/Control 

Measure 

1 

Separate  Systems 

1 

1 

2 

Shared  Resources 

1 

2 

3 

Gateways 

2 

3 

4 

Multiple  Entry  Points 

2 

4 

5 

Conformable/Compatible 

Systems 

3 

4 

6 

Completely  Interoperable 

Systems 

3 

5 

7 

Same  System 

4 

6 

Quantification  of  Interoperability  Methodology  (QoIM) 

In  1989,  Dennis  Mensh,  Robert  Kite,  and  Paul  Darby  published  a  paper  in  the  Naval  Engineer’s 
Journal  called  “The  Quantification  of  Interoperability.”  For  ease  of  reference,  we  nickname  this 
model  “QoIM.”  Messrs.  Mensh  and  Kite  worked  for  MITRE  Corporation  at  the  time  this  article 
was  published  and  may  have  laid  some  of  the  groundwork  for  the  well-known  LISI  model  which 
was  published  by  MITRE  nine  years  later  although  they  were  never  cited  as  a  source  in  the  LISI 
paper.  Mensh,  Kite,  and  Darby’s  approach  to  interoperability  measurement  published  in  1989  is 
unique.  Their  specific  goal  was  to  create  a  means  of  “(1)  assessing  interoperability  issues  for 


three  mission  areas:  wide  area  surveillance  (WAS),  over-the-horizon  targeting  (OTH-T),  and 
electronic  warfare  (EW);  and  (2)  quantifying... seven  interoperability  components.”  [Mensh,  Kite, 
&  Darby,  1989,  p.  252]  They  stated  that  “interoperability  of  systems,  units,  or  forces  can  be 
factored  into  a  set  of  components  that  can  quantify  interoperability.”  [Ibid,  p.  251]  They 
identified  and  defined  seven  components  as  media,  languages,  standards,  requirements, 
environment,  procedures,  and  human  factors.  [Ibid,  pp.  253-254]  They  defined  a  Measure  of 
Effectiveness  (MOE)  logic  function  for  each  component  and  used  that  logic  function  to  create  a 
truth  table  for  each  component.  For  example,  the  MOE  logic  function  for  the  Language 
component  was  defined  as  “Message  Correctness  =  Intelligibility  and  Manual  Intervention  & 
Error.”  [Ibid,  pp.  255-256]  The  truth  table  listed  the  binary  MOE  value  (e.g.,  Message 
Correctness,  Intelligibility,  and  Manual  Intervention  and  Error)  for  various  “significant  events” 
which  occurred  during  an  exercise  or  simulation.  The  truth  table  could  be  analyzed — the 
presence  of  zeros  indicated  lack  of  interoperability  during  certain  component  events  and  the 
presence  of  ones  indicated  some  level  of  interoperation  occurred.  [Ibid,  pp.  254-255]  In  the  end, 
a  final  Interoperability  Data  Table  was  formed  showing  the  truth  table  results  for  all  seven 
interoperability  components.  Rather  than  provide  a  final  measure  for  interoperability,  this  table 
gives  seven  ratios  (one  for  each  component)  which  show  the  number  of  significant  events  scoring 
a  one  divided  by  the  total  number  of  events.  Mensh,  Kite,  and  Darby  state  three  benefits  of  this 
table:  “(1)  It  illustrates  the  overall  quantification  of  interoperability.  (2)  For  specific  events  it 
enables  an  evaluation  of  the  interoperability  of. .  .systems  in  terms  of  the  seven  interoperability 
components  in  terms  of  the  corresponding. .  .events.  (3)  Having  this  type  of  table  for  two 
different. .  .architectures  enables  a  comparison  of  the  relative  goodness  of  each  architecture.” 

[Ibid,  p.  259]  Although  Mensh,  Kite,  and  Darby  state  in  their  paper  that  their  “methodology  for 
quantifying  interoperability  is  being  pursued,”  they  also  state  that  “additional  exercises  will  be 
required  and  are  currently  in  the  planning  stages.”  [Ibid]  Aside  from  one  citation  by  Leite  in 
1998  (and  revised  paper  in  2003),  we  have  been  unable  to  find  any  further  mention  or  use  of  this 
model  beyond  the  original  journal  article  in  which  it  was  proposed.  [Leite,  2003] 

Military  Communications  and  Information  Systems  Interoperability  (MCISI) 

Another  interesting  interoperability  measurement  model  was  published  in  1 996  which 
was  designed  to  model  communications  and  information  systems  (CIS)  interoperability  in  a 
mathematical  way.  Based  upon  the  title  of  the  article  the  methodology  was  introduced  in,  we 
give  it  the  acronym  MCISI.  Amanowicz  and  Gajewski  recognized  that  “a  great  amount  of  data  is 
needed  as  well  as  the  appropriate  methodology  of  interoperability  modeling.”  [Amanowicz  and 
Gajewski,  1996,  p.  281]  They  stated  that  interoperability  modeling  is  “a  multistaged  process 
which  combines  operational  requirements,  CIS  data,  standards,  interfaces  and  modeling 
facilities.”  [Ibid]  They  mention  that  “the  results  of  interoperability  modeling  are  most  often 
presented  in  the  (sic)  three-level  scale  starting  from  full  through  partial  to  non-interoperable 
systems  for  particular  services  (operational  procedures).”  [Ibid,  p.  282]  They  use  a  colored  cube 
to  visualize  the  model  in  which  one  axis  is  level  of  command,  the  second  is  CIS  services,  and  the 
third  is  transmission  medium.  The  color  of  the  intersections  is  red,  yellow,  or  green  representing 
none,  partial,  or  full  interoperability  of  a  specific  service  through  a  specific  medium  at  a  specified 
level  of  command.  The  mathematics  of  their  methodology  comes  next  as  Amanowicz  and 
Gajewski  describe  a  set  of  systems  as  “points  in  multi-dimensional  space”  and  the  features  of 
these  systems  as  “coordinates  of  these  points.”  [Ibid]  They  then  define  the  normalized  “distance” 

zh-6,'1 

between  two  points  as  d  ( A,  B)  =  — - ,  where  a, ,  bt  are  features  of  system  A  and  system  B, 

n 

respectively.  [Ibid]  They  state  that  when  d(T,,8)  =  0  systems  A  and  B  achieve  full 


interoperability  and  when  d  (A, B)  >  1 ,  the  system  pair’s  interoperability  decreases.  [Ibid] 

Amanowicz  and  Gajewski  then  expand  this  model  to  accommodate  a  set  of  Z  systems  by  creating 
dendrite  (a  broken  line  which  connects  all  points  of  a  set)  arrangements  of  the  systems.  [Ibid] 
They  state  that  the  best  arrangement  is  the  one  with  the  shortest  dendrite  length.  [Ibid,  p.  283] 
While  their  abridged  paper  could  not  fully  provide  all  the  details  of  their  method,  they  enjoy  the 
distinction  of  being  the  first  to  recognize  that  the  upper/lower  limits  of  interoperability  of  a  given 
set  of  systems  can  be  measured  and  to  infer  that  optimization  can  be  performed  in  order  to 
determine  the  best  arrangement  of  systems  in  order  to  maximize  interoperability.  [Ibid] 
Unfortunately,  like  SoIM  and  QoIM,  we  were  unable  to  find  proof  of  any  further 
institutionalization  of  Amanowicz’  and  Gajewski’s  MC1S1  model  after  publication. 

Levels  of  Information  System  Interoperability  (LISI)  Model 

L1SI  model  development  began  under  contract  at  the  MITRE  Corporation  in  1993  and 
was  published  in  final  form  by  the  C4ISR  Architecture  Working  Group  (AWG)  in  1998.  The 
AWG  was  co-chaired  by  the  Joint  Staff  J6I  and  the  Director,  Architectures  Directorate  of  the 
C41SR  Integration  Support  Activity  (CISA)  which  was  under  the  direction  of  OSD(ASD(C3I)). 
[C4ISR  AWG,  Apr.  1998,  p.  5]  In  their  report  on  LISI,  the  AWG  stated  “We  lack  a  practical 
assessment  process  for  determining  the  interoperability  maturity  level  or  ‘metric’  of  a  given 
system  or  system  pair. .  .The  LISI  Assessment  Process,  with  its  associated  tool,  system  profiles, 
and  data  repository,  fills  these  needs.”  [C4ISR  AWG,  Mar.  1998,  p.  ES-7]  Thus,  LISI  is  system 
or  system  pair- focused  vice  mission,  scenario,  or  operational  thread  focused.  The  LISI  report 
also  states  that  LISI  is  a  “process  for  defining,  evaluating,  measuring,  and  assessing  information 
systems  interoperability,”  implying  that  LISI  is  best  suited  for  measuring  information  systems 
interoperability.  [IBID,  p.  1-2] 

While  CJCSI  6212. 01C,  Interoperability  and  Supportability  of  Information  Technology 
and  National  Security  Systems,  required  program  managers  to  ensure  that  they  complied  with  a 
range  of  LISI  profile  requirements,  the  latest  version  of  that  document,  CJCSI  62 12.0  ID,  “deletes 
requirements  associated  with  Levels  of  Information  Systems  Interoperability  (LISI).”  [CJCSI 
61212.01C,  2003,  p.  A-6],  [CJCSI  6212.01D,  2006,  p.  5]  Although  the  AWG  had  originally 
stated  a  goal  of  having  LISI  institutionalized  in  DoDD  4630.5,  DoDI  4830.8,  DoDI  5000.2,  and 
DoDD  5000.1,  our  review  of  all  versions  of  these  documents  published  since  1998  indicates  no 
mention  of  LISI.  However,  the  Joint  Chiefs  of  Staff  did  maintain  a  repository  of  LISI  profiles  for 
acquisition  programs  as  a  result  of  the  CJCSI  62 12.0 1C  requirement  (on  classified  SIPRNet 
http://lisi.ncr.disa.smil.mil).  [CJCSI  6212. 01C,  2003,  p.  K-2] 

As  this  model  has  been  described  in  numerous  technical  reports,  articles,  and  papers  over 
the  years,  it  will  be  summarized  very  briefly  and  references  to  more  detailed  summaries  will  be 
provided.  Suffice  it  to  say  that  LISI  has  been  acknowledged  as  the  most  prominent 
interoperability  measurement  model  within  the  Department  of  Defense  over  the  past  decade. 

The  LISI  model  is  comprised  of  the  LISI  Assessment  Basis  and  LISI  Assessment 
Products.  [C4ISR,  1998,  p.  1-4]  The  Assessment  Basis  includes  an  Interoperability  Maturity 
Model,  a  Reference  Model,  a  Capabilities  Model,  and  Implementation  Options  Tables.  [Ibid]  The 
Assessment  Products  include  Interoperability  Profiles,  Interoperability  Metrics,  Interoperability 
Matrices,  Comparison  Tables,  and  Architecture  Products. 

LISI  Assessment  Basis.  The  LISI  Interoperability  Maturity  Model  is  similar  to 
LaVean’s  SoIM  in  that  it  describes  levels  of  interoperability.  Whereas  LaVean  had  seven  levels, 
LISI  has  five  levels  called  maturity  levels.  Level  0  is  Isolated,  Level  1  is  Connected,  Level  2  is 
Functional,  Level  3  is  Domain,  and  Level  4  is  Enterprise.  [C4ISR,  1998,  pp.  2-6  -  2.7]  However, 
unlike  LaVean’s  SoIM,  LISI  makes  an  improvement  step  by  describing  four  attributes  within 
each  level  which  “encompass  the  full  range  of  interoperability  considerations.”  [Ibid]  The  four 


attributes  are  described  by  the  acronym  PAID — Procedures,  Applications,  Infrastructure,  and 
Data.  The  LISI  Reference  Model  puts  the  five  levels  on  the  rows  of  a  matrix  and  the  four 
attributes  on  the  columns.  The  cells  at  the  intersections  describe  the  capabilities  needed  to 
achieve  a  specified  level  of  interoperability  for  a  certain  attribute.  [Ibid,  p.  3-1]  The  LISI 
Capabilities  Model  is  a  “decomposition”  of  the  reference  model  which  describes  specific 
capabilities  required  to  achieve  a  specified  level  of  interoperability  for  a  certain  attribute.  [Ibid,  p. 
3-12]  The  LISI  Implementation  Options  Tables  are  used  to  in  building  a  systems’ 

Interoperability  Profile — one  of  the  LISI  Assessment  Products.  [Ibid,  pp  3-17] 

LISI  Assessment  Products.  The  first  LISI  product  created  during  the  LISI 
interoperability  assessment  process  is  the  system  Interoperability  Profile.  This  document  is 
created  by  answering  a  web-based  questionnaire  which  gathers  information  about  a  system  for  all 
four  interoperability  attributes.  [Ibid]  Once  the  questionnaire  is  complete,  an  Interoperability 
Metric  can  be  obtained.  This  metric  is  a  triplet  of  metric  type  (Generic,  Expected,  &  Specific), 
Level  (0. .  .4),  and  Sub-level  (a. .  .z).  The  metric  quantitatively  describes  the  level  of 
interoperability  for  one  system  (generic)  or  a  pair  of  systems  (expected  and  specific).  [Ibid,  p.  4- 
4]  The  generic  metric  is  the  highest  level  of  interoperability  a  single  system  is  capable  of 
whereas  the  expected  metric  describes  the  highest  common  level  of  interoperability  between  two 
systems.  The  specific  metric  describes  the  highest  common  level  of  interoperability  between  two 
information  systems  across  all  PAID  attributes.  [Ibid]  The  Interoperability  Matrix  takes  the 
Interoperability  Metrics  for  multiple  systems  and  arrays  them  on  a  grid  in  which  systems  are 
labeled  on  the  rows  and  columns  and  the  intersections  contain  the  interoperability  level  for  that 
system  pair.  The  color  of  the  intersections  indicates  whether  the  specific  interoperability  for  the 
system  pairs  exceeds,  equals,  or  is  less  than  the  expected  level  for  the  pair.  [Ibid,  p.  4-10]  Thus, 
the  matrix  can  be  used  to  visualize  the  interoperability  of  a  group  of  systems.  Comparison  Tables 
can  be  used  to  “provide  a  comparison  of  interoperability  implementation  information  between 
systems  in  terms  of  PAID,”  and  are  flexible  in  that  they  can  be  used  for  a  specific  attribute,  or 
more  generically  for  a  group  of  attributes.  [Ibid,  p.  4-1 1]  Finally,  various  architecture  products 
can  be  created  using  LISI  assessment  data.  For  example,  FISI  generic  metrics  for  individual 
systems  can  be  overlaid  upon  a  system  interface  description,  thus  giving  not  only  a  visualization 
of  the  connectivity  of  systems,  but  of  the  quality  of  connectivity  of  those  systems.  [Ibid,  p.  4-16] 

FISI  has  been  reviewed  and  critiqued  by  many  other  researchers  since  its  publication. 
Several  recent  reviews  have  been  done  by  Brownsword,  et  al.,  Carney  &  Obemdorf ,  Clark  & 
Jones,  Clark  &  Moon,  Kasunic  &  Anderson,  Morris,  et  al.,  Tolk,  and  others.  [Brownsword,  et  al., 
2004],  [Carney  &  Obemdorf,  2004],  [Clark  &  Jones,  1999],  [Clark  &  Moon,  2001],  [Kasunic  & 
Anderson,  2004],  [Morris,  et  al.,  2004],  [Tolk,  2003] 

Interoperability  Assessment  Methodology  (IAM) 

The  Interoperability  Assessment  Methodology  (we  call  it  IAM)  was  published  in  the 
proceedings  of  the  66th  Military  Operations  Research  Society  (MORS)  Symposium  three  months 
after  FISI  was  published  and  was  revised  again  in  1999  and  2003.  It  is  unknown  if  the  author, 
Michael  Feite,  of  SIM,  Inc.  was  aware  of  the  FISI  effort,  however,  of  the  fourteen  models 
surveyed,  he  was  only  author  to  reference  the  Mensh,  Kite,  and  Darby  Quantification  of 
Interoperability  Model  (QoIM)  in  his  paper.  Fike  QoIM,  IAM  is  based  upon  the  idea  of 
“measurement  and  quantification  of  a  set  of  interoperability  system  components.”  [Feite,  2003,  p. 
1]  Mr.  Feite  identified  nine  components  (vice  QoIM’s  seven)  which  are  requirements,  standards, 
data  elements,  node  connectivity,  protocols,  information  flow,  latency,  interpretation,  and 
information  utilization.  [Ibid,  p.  3]  Each  of  the  nine  components  has  either  a  “yes/no”  answer  or 
a  mathematical  equation  associated  with  it  [Ibid,  p.  22].  Mr.  Feite  also  defines  “degrees  of 
interconnection”  which  are  connectivity,  availability,  interpretation,  understanding,  utility, 
execution,  and  feedback.  [Feite,  pp.  3-8]  He  summarizes  his  Interoperability  Assessment  process 


in  the  form  of  a  flowchart  and  applies  the  process  to  the  Navy’s  Tactical  Ballistic  Missile  Defense 
Program  as  an  example.  Mr.  Leite’s  methodology  also  does  not  appear  to  have  been 
institutionalized,  but  was  referenced  by  Kasunic  and  Anderson  in  their  2004  Tech  Note  in  which 
they  graphically  show  that  a  “Mission  Slice”  can  include  a  color-coded  version  of  L1SI  metrics 
for  systems  supporting  the  slice  and  they  further  state  that  Mr.  Leite’s  interoperability  “quality 
attributes”  can  be  used  to  extend  the  LISI  model  at  this  mission  slice  level.  [Kasunic  &  Anderson, 
2004,  p.  26] 


Organisational  Interoperability  Maturity  Model  for  C2  (OIM) 

In  1998,  the  Australian  Defense  Science  and  Technology  Organisation  (DSTO) 
completed  a  Command  and  Control  Support  (C2S)  study.  [Clark  &  Jones,  1999,  p.  1]  In  this 
study,  they  described  five  layers  of  C2  Support  (Telecommunications,  Info  Technology,  Info 
Management,  C2  Process,  and  C2  Framework),  but  they  also  determined  that  the  just  published 
LISI  model  could  not  easily  map  onto  their  model  for  three  main  reasons,  1)  LISI  is  strongly 
technological,  2)  LISI  focuses  on  system  and  technical  compatibility,  and  3)  LISI  does  not 
address  higher  layers  of  C2  support.  [Ibid,  p.  4]  As  a  result,  Clark  and  Jones  determined  to  create 
an  extension  to  LISI  to  “cover  the  organizational  aspects  of  interoperability.”  [Ibid]  The  results 
of  their  labors  is  the  Organisational  Interoperability  Maturity  Model  (OIM)  first  introduced  in 
June  1999  at  the  International  Command  and  Control  Research  and  Technology  Symposium,  then 
revised  in  2003  at  the  same  conference  by  Fewell  and  Clark.  [Ibid],  [Fewell  &  Clark,  2003] 

The  OIM  model  was  used  to  “identify  problems  and  evaluate  interoperability  in  a 
coalition  operation.”  [Ibid,  2003,  p.  3]  It  was  specifically  designed  as  an  extension  to  LISI. 

[Clark  &  Jones,  1999,  p.  4]  Just  as  LISI  defined  five  levels  of  interoperability,  so  did  OIM 
(independent,  cooperative,  collaborative,  combined,  and  unified).  [Ibid,  p.  6],  [Fewell  &  Clark, 
2003,  p.  3]  Whereas  LISI  defined  four  attributes  of  interoperability  described  by  the  PAID 
acronym,  Clark  and  Jones  defined  four  attributes  of  organizational  interoperability.  These  are  1) 
preparation,  2)  understanding,  3)  command  and  coordination,  and  4)  ethos  (Socio-Cultural 
factors).  [Ibid,  p.  8,  10]  Fewell  and  Clark  supplied  detailed  descriptions  of  the  attributes, 
identified  multiple  sub-attributes  for  each  of  the  four  main  attributes  and  used  the  revised  model 
to  analyze  the  operational  interoperability  three  scenarios:  1)  the  multi-national  force 
participating  in  the  Australian  led,  1999-2000  International  Force  East  Timor  (INTERFET) 
operation,  2)  an  AS-US  interoperability  review,  and  3)  the  Multinational  Limited  Objective 
Experiment  2  (MNLOE2)  held  in  February  2003.  [Ibid,  pp.  4-14]  These  applications  of  the  OIM 
showed  that  the  model  is  used  in  the  same  fashion  as  LISI,  and  indeed  extends  the  LISI  model  in 
an  appropriate  and  useful  fashion.  It  is  our  opinion  that  the  combined  LISI-OIM  model  is  a  most 
valuable  tool  for  performing  basic,  high-level  interoperability  analysis  of  information  systems 
used  by  operational  forces  in  a  specific  scenario.  The  OIM  model  was  reviewed  by  several 
researchers  since  its  initial  introduction  in  1999.  Some  examples  are:  Briscombe,  et  al., 
Brownsword,  et  al.,  Clark  &  Jones,  Clark  &  Moon,  Fewell,  et  al.,  Kasunic  &  Anderson,  and 
Morris,  et  al.  [Briscombe,  et  al.  2006],  [Brownsword,  et  al.,  2004],  [Clark  &  Jones,  1999],  [Clark 
&  Moon,  2001],  [Fewell,  et  al.,  2004],  [Kasunic  &  Anderson,  2004],  [Morris,  et  al.,  2004] 
Additionally,  another  researcher,  Anthony  Dekker,  has  also  published  a  modification  to  the  model 
which  he  uses  a  modification  of  the  OIM  model  to  analyze  the  Black  Hawk  Down  incident  in 
Mogadishu  in  1993.  [Dekker,  2005]  It  is  unknown  whether  the  OIM  model  has  been 
institutionalized  by  the  Australian  Department  of  Defence. 


Stoplight 


In  2002,  Hamilton,  Rosen,  and  Summers  published  an  Interoperability  Roadmap  for 
C4ISR  Legacy  Systems  which  included  a  very  uncomplicated  interoperability  measurement 
model  which  they  simply  called  a  Stoplight  model.  [Hamilton,  Rosen  &  Summers,  2002,  p.  17, 
21]  The  creators  of  the  Stoplight  model  state  that  “interoperability  is  notoriously  difficult  to 
measure,”  yet  still  propose  a  “simplified  model”  to  measure  it.  [Ibid,  pp.  20-21]  Their  model 
turns  out  to  be  quite  useful  (in  its  restricted  sphere  of  applicability)  as  a  starting  point  for 
interoperability  analysis  in  spite  of  its  simplicity.  The  model’s  purpose  is  to  help  decision  makers 
understand  whether  or  not  their  legacy  systems  meet  operational  and  acquisition  interoperability 
requirements.  [Ibid,  p.  21]  The  model  is  designed  as  a  two-dimensional  matrix  in  which  “meets 
operational  requirements  (yes/no)”  appears  on  the  rows  of  the  matrix  and  “meets  acquisition 
requirements  (yes/no)”  appears  on  the  columns.  [Ibid]  The  intersections  of  the  matrix  are  colored 
red,  yellow,  orange,  and  green  depending  on  how  well  the  specific  type  of  requirement  is  met. 
[Ibid]  Definitions  of  the  color  scheme  are  provided  in  Hamilton,  Rosen,  and  Summers’  paper. 
[Ibid,  p.  22]  The  authors’  also  give  an  example  of  how  a  time-line  with  the  color-coding  overlaid 
can  be  created  to  show  the  plan  to  achieve  improved  interoperability  in  the  future.  [Ibid,  p.  23] 

We  have  found  no  evidence  that  this  model  was  institutionalized  within  the  Department  of 
Defense. 


Levels  of  Conceptual  Interoperability  Model  (LCIM) 

The  Levels  of  Conceptual  Interoperability  Model  (LCIM)  was  published  by  Andreas 
Tolk  and  James  Muguira  in  2003  with  the  intent  that  it  be  used  to  “become  a  bridge  between  the 
conceptual  design  and  the  technical  design  for  implementation,  integration,  or  federation,”  and 
that  it  be  used  to  “enhance  the. .  .DoD  Net-Centric  Data  Strategy  for  the  Global  Information  Grid 
(GIG).”  [Tolk,  2003,  p.  1]  Additionally,  Tolk  &  Muguira  state  that  it  can  be  used  as  a  framework 
“to  determine  in  the  early  stages  of  the  federation  development  process  whether  meaningful 
interoperability  between  systems  is  possible.”  [Ibid]  Tolk  &  Muguira  focus  their  model  on  the 
world  of  modeling  and  simulation,  and  similar  to  LISI  and  OIM,  proffer  five  levels  of 
interoperability  geared  specifically  toward  conceptual  interoperability.  They  are  Level  0 — 
System  Specific  Data,  Level  1 — Documented  Data,  Level  2 — Aligned  Static  Data,  Level  3 — 
Aligned  Dynamic  Data,  and  Level  4 — Harmonized  Data.  [Ibid,  pp.  2-3]  The  number  of  levels 
was  eventually  extended  to  seven,  and  the  names  were  changed  as  well  due  to  “new  research  at 
VMASC  and  as  the  response  to  critique  by  the  scientific  community.”  [Tumitsa  &  Tolk,  2006,  p. 
1]  The  final  levels  are  Level  0 — No  interoperability,  Level  1 — Technical  interoperability,  Level 
2 — Syntactic  Interoperability,  Level  3 — Semantic  Interoperability,  Level  4 — Pragmatic 
Interoperability,  Level  5 — Dynamic  Interoperability,  and  Level  6 — Conceptual  Interoperability. 
[Tumitsa  &  Tolk,  2006,  p.  2]  LCIM  is  flexible  enough  to  allow  a  developer  to  define  metrics 
appropriate  to  the  level  of  alignment  needed  between  two  or  more  models,  and  Tolk  &  Muguira 
aptly  make  the  point  that  different  models  require  different  levels  of  interoperability — in  other 
words,  level  4  is  not  the  goal  for  all  situations.  [Tolk  &  Muguira,  p.  5]  Tolk  &  Muguira’s  model 
makes  clear  that  “meaningful  interoperability  of  simulation  systems  on  the  implementation 
level. .  .requires  composability  of  the  underlying  conceptual  models.”  [Ibid,  p.  9]  LCIM  seems  to 
be  receiving  considerable  attention  within  the  modeling  and  simulation  community  and  has  been 
widely  cited.  Others  who  have  reviewed  LCIM  include  Brownsword,  et  al.,  Kasunic  & 

Anderson,  and  Morris,  et  al.  [Brownsword,  et  al.,  2004],  [Kasunic  &  Anderson,  2004],  [Morris,  et 
al,  2004] 


Layers  of  Coalition  Interoperability  (LCI) 


In  2003,  the  same  Andreas  Tolk  who  introduced  the  LCIM  model  also  introduced  a 
different,  but  similarly  acronymed.  Layers  of  Coalition  Interoperability  (LCI)  model.  While  he 
states  that  this  model  is  “in  its  infancy,”  he  hopes  that  it  will  become  a  “hub  for  future  work.” 
[Tolk,  2003,  p.  21]  LCI  defines  nine  layers  of  interoperability,  and  shows  through  his  reference 
model  that  there  is  a  continuum  between  technical  interoperability  and  operational 
interoperability  rather  than  a  distinct  breakpoint  between  the  two.  [Ibid,  p.  18]  While  others  have 
described  separate  models  for  technical  and  operational  interoperability,  Dr.  Tolk  demonstrates 
that  the  interface  between  technical  and  operational  interoperability  is  made  at  the 
knowledge/awareness  layer.  [Ibid]  The  nine  layers  in  Dr.  Tolk’s  LCI  model  are,  from  lowest  to 
highest,  1)  Physical  Interoperability,  2)  Protocol  Interoperability,  3)  Data/Object  Model 
Interoperability,  4)  Information  Interoperability,  5)  Knowledge/Awareness,  6)  Aligned 
Procedures,  7)  Aligned  Operations,  8)  Harmonized/Strategy  Doctrines,  and  9)  Political 
Objectives.  These  layers  are  framed  by  a  “common  model  of  the  operation.”  [Ibid,  p.  12]  Dr. 
Tolk  proposes  possible  metrics  for  his  model  as  those  contained  in  the  NATO  Code  of  Best 
Practice  for  C2,  Code  of  Best  Practice  for  Experimentation,  and  Network  Centric  Warfare 
Metrics  Framework.  [NATO,  2002],  [Alberts,  et  al.,  2002],  [Alberts,  Garstka,  &  Stein,  2000] 
Interestingly,  LISI  and  NMI  were  referenced  by  Dr.  Tolk,  but  OIM  (which  was  published  four 
years  earlier,  also  in  the  ICCRTS)  was  not.  Dr.  Tolk  states  in  his  paper  that  LCI  is  not  meant  to 
be  a  “universal  replacement”  for  other  frameworks,  but  is  meant  to  be  used  to  “help  formulate 
layered  models.”  [Tolk,  2003,  p.  17]  LCI  has  been  cited  and  briefly  reviewed  by  one  other  group 
that  we  know  of,  Morris,  et  al.  [Morris,  et  al.,  2004] 

NATO  C3  Technical  Architecture  Reference  Model  for  Interoperability  (NMI) 

Version  four  of  this  NATO  reference  model  was  published  in  March  2003  and  according 
to  Morris,  et  al.,  it  was  updated  to  closely  reflect  the  LISI  model  in  December  2003.  It  is  no 
longer  available  on  the  NATO  website.  NMI  originally  described  four  degrees  of  interoperability 
(not  including  degree  0  which  was  no  interoperability).  The  four  degrees  were:  1)  unstructured 
data  exchange,  2)  structured  data  exchange,  3)  seamless  sharing  of  data,  and  4)  seamless  sharing 
of  information.  These  degrees  (including  degree  0)  map  directly  to  LISI’s  five  levels  of 
interoperability.  NMI  was  overviewed  by  Brownsword,  et  al.,  Kasunic  &  Anderson,  Morris,  et 
al.,  Tolk  &  Muguira,  and  Tolk.  [Brownsword,  et  al.,  2004],  [Kasunic  &  Anderson,  2004], 

[Morris,  et  al,  2004],  [Tolk  &  Muguira,  2003],  [Tolk  2003] 

System-of-Systems  Interoperability  (SoSI)  Model 

This  simple  model  was  published  in  2004  by  the  Carnegie-Mellon  University  Software 
Engineering  Institute  (CMU-SEI)  by  Morris,  et  al.  According  to  the  CMU-SEI  on-line 
Interoperability  Guide,  SoSI  was  developed  to  enable  CMU-SEI  researchers  to  more  effectively 
pursue  system-of-systems  interoperability  research.  [CMU-SEI]  The  SoSI  model  is  founded 
upon  three  types  of  interoperability  (operational,  constructional,  and  programmatic)  and  the 
activities  associated  with  each.  [Ibid]  While  it  appears  to  be  a  useful  way  of  developing  and 
integrating  systems-of-systems,  SoSI  lacks  metrics  to  specifically  measure  interoperability  within 
a  system-of-system.  For  this  reason,  we  considered  not  including  it  in  this  survey,  but  in  the  end 
decided  to  include  it  because  it  provides  a  framework  in  which  an  analyst  can  use  his/her  own 
metrics  to  measure  system-of-systems  interoperability.  The  technical  report  in  which  SoSI  is 
introduced  contains  a  summary  of  LISI,  OIM,  NMI,  LCIM,  LCI,  and  SoSI.  Morris,  et  al.,  also 
include  in  their  paper  a  useful  listing  of  various  DoD  interoperability  initiatives.  We  found  no 


evidence  that  SoSI  has  been  institutionalized  within  the  DoD  or  further  referenced  outside  of 
CMU-SEI. 


Non-Technical  Interoperability  (NTI)  Framework 

Stewart,  et  al.,  introduced  the  Non-Technical  Interoperability  (NTI)  framework  in  2004 
for  the  purpose  of  developing  a  “valid  framework  describing  the  factors  that  undeipin  NTI” 
which  would  allow  the  United  Kingdom’s  Ministry  of  Defence  “to  understand  these  aspects  of 
interoperability  better  and  to  mitigate  potential  frictional  factors  in  multinational  forces.” 

[Stewart,  et  al.,  2004]  Stewart,  et  al.  felt  that  the  DSTO  OIM  model  was  a  “useful  top-level 
framework”  for  the  data  they  captured  in  their  own  research.  [Stewart,  et  al.,  2004,  p.  6]  But  they 
also  recognized  that  using  the  term  operational  interoperability  left  out  other  factors  such  as 
“social,  personnel,  and  process  factors,”  so  they  decided  to  use  the  term  non-technical 
interoperability  in  order  to  be  more  comprehensive.  [Ibid]  They  did  not,  however,  reference  any 
of  the  other  non-technical  interoperability  models  already  published  such  as  LCI.  The  four 
enabling  attributes  (preparedness,  understanding,  command  style,  and  ethos)  originally  proposed 
by  Clark  and  Jones  in  their  OIM  model  form  the  core  of  the  NTI  framework  and  the  research 
performed  by  Stewart,  et  al.  provided  a  more  detailed  breakdown  of  these  attributes.  [Clark  & 
Jones,  1999,  p.  8],  [Stewart,  et  al.,  2004,  p.  6]  While  a  complete  set  of  metrics  was  not  provided 
by  Stewart,  et  al.,  they  did  propose  a  Multinational  Forces  Co-operability  Index  which  provides  a 
score  of  1,  2,  4,  8,  12,  or  16  for  two  (preparedness  and  understanding)  of  the  four  attributes.  [Ibid, 
pp.  8-9]  For  this  reason  we  included  NTI  as  an  interoperability  measurement  model  in  our 
survey.  While  the  NTI  framework  was  developed  as  result  of  45  interviews  with  UK  military 
officers  ranging  in  rank  from  Army  Captain  to  3 -star  General,  it  is  unknown  if  the  framework  is 
undergoing  further  refinement  or  has  yet  to  have  been  institutionalized  within  the  UK  Ministry  of 
Defence.  [Stewart,  et  al,  2004,  pp.  5-6] 

Organisational  Interoperability  Agility  Model  (OIAM) 

Kingston,  Fewell,  and  Richer  of  the  Australian  Defence  Science  and  Technology 
organization  (DSTO)  published  the  Organisational  Interoperability  Agility  Model  (OIAM)  in 
2005.  According  to  its  authors,  it  “builds  on  the  organizational  Interoperability  Model  developed 
by  Clark  and  Jones”  and  “aims  to  capture  the  dynamic  aspects  of  working  in  coalitions  including 
the  ability  of  an  organization  to  contribute  to  the  rapid  formation  and  reformation  of  coalitions, 
including  novel  ones.”  [Kingston,  Fewell,  and  Richer,  2005,  p.  2]  Organizational  agility  is 
defined  by  Kingston,  Fewell,  and  Richer  as  “a  single  organisation’s  potential  to  have  agile 
interfaces  to  other  organizations  in  future  coalition  operations”  and  “assesses  an  organisation’s 
ability  to  adapt  to  changing  circumstances.”  [Ibid,  p.  3]  OIAM  began  with  the  idea  of  creating  a 
maturity  model  similar  to  OIM  and  decided  to  use  five  levels  of  organizational  agility  in  order  to 
more  closely  align  with  OIM.  Additionally,  OIAM  makes  use  of  the  four  OIM  attributes,  only 
combines  preparation  and  understanding.  Therefore,  the  five  levels  are:  Level  0 — Static,  Level 
1 — Amenable,  Level  2 — Accomodating,  Level  3 — Open,  and  Level  4 — Dynamic.  The  three 
attributes  are  1)  Preparation  +  Understanding,  2)  Command  and  coordination,  and  3)  Ethos.  [Ibid, 
pp.  12-15]  Specific  definitions  of  these  levels  and  attributes  are  provided  by  Kingston,  Fewell, 
and  Richer  in  order  to  help  the  analyst  properly  rate  an  organization’s  agility.  [Ibid]  Although  no 
specific  metrics  are  mentioned,  we  believe  the  model’s  levels  themselves  are  metrics  and  can  be 
used  to  form  generic,  specific,  and  expected  metrics  similar  to  those  in  LISI  and  OIM.  The 
model’s  developers  indicate  that  they  are  at  the  beginning  of  their  research  on  organizational 
agility  and  that  they  plan  to  develop  additional  metrics  and  perform  case  studies  in  order  to  refine 
the  model.  [Ibid,  p.16]  As  a  new  model,  it  has  not  yet  been  institutionalized  by  the  Australian 


Department  of  Defence.  OIAM  does  not  yet  appear  to  have  been  referenced  by  any  other 
published  works. 


The  Layered  Interoperability  Score  ( i-Score ) 

The  Layered  Interoperability  Score  ( i-Score )  is  a  new  methodology  created  by  the  same 
authors  who  have  written  this  survey  on  interoperability  measurement.  It  has  been  submitted  and 
accepted  for  publication  in  the  Proceedings  of  the  2007  Conference  on  Systems  Engineering 
Research  (CSER  2007).  [Ford,  Colombi,  Graham,  &  Jacques,  2007,  p.  1]  i-Score  is  a 
mathematical  method  of  measuring  the  interoperability  of  all  types  of  systems  (technological, 
biological,  organizational,  and  environmental).  [Ibid]  It  is  easily  computed,  based  upon  an 
operational  thread,  makes  use  of  existing  architecture  data,  can  be  used  in  scenarios  where  more 
than  one  type  of  interoperability  is  included,  and  provides  a  means  of  quantitatively  measuring 
the  interoperability  of  the  systems  supporting  an  operational  thread.  [Ibid,  p.  2]  Unique  to  the  i- 
Score  methodology  is  a  means  of  determining  a  realistic  theoretical  upper  limit  on 
interoperability  for  the  systems  supporting  the  operational  thread.  [Ibid]  The  methodology  can 
quickly  determine  ways  to  close  the  “interoperability  gap”  between  the  as-is  i-Score  and  the 
theoretical  optimum  i-Score.  [Ibid,  p.  6]  The  i-Score  interoperability  measurement  methodology 
is  also  unique  in  that  it  can  make  use  of  custom  layers  (matrices)  which  allow  the  analyst  to 
compensate  the  i-Score  measurement  for  any  number  of  interoperability-related  factors  such  as 
bandwidth,  protocols,  mission  capability  rate,  probability  of  connection,  atmospheric  effects,  etc. 
It  is  even  possible  to  create  cost,  schedule,  reliability,  and  performance  layers  to  measure  the 
impact  of  various  programmatic  changes  on  the  interoperability  of  the  thread.  [Ibid,  p.  4]  Since 
the  methodology  is  entirely  mathematical,  it  is  possible  to  embed  custom  functions  of 
interoperability-related  variables  so  that  optimization  can  be  performed  to  determine  the  best 
“settings”  for  maximum  interoperability.  The  methodology  can  be  used  to  make  non-traditional 
interoperability  measurements  such  as  organizational  or  policy  interoperability  measurements. 

The  i-Score  interoperability  measurement  methodology  is  a  six-step  process.  [Ibid,  pp.  3- 
6]  The  six  steps  of  the  methodology  are  described  below. 


Step  1 — Diagram  the  operational  thread  (e.g.,  time-critical-targeting)  using  an  IDEF0,  BPML,  or 
UML  activity  diagram  and  define  the  ordered  set  T  of  systems  supporting  each  activity  in  the 
thread.  Step  2 — Create  an  interoperability  matrix  M  =  \  cirsu  1  where  n  is  the  number  of 

L  J  V  J  nxn 

systems  supporting  the  thread,  C  =  [ciy  ]nxn  is  a  multiplicity  matrix  which  describes  the  number  of 
times  a  system  is  used  in  the  thread,  and  S  =  [v(/  ]nxn  is  a  spin  matrix  where  .v,7  e  {-1,0,1}  is  a 


variable  indicating  no  human  or  machine  translation  needed  for  a  system  pair  (+1),  machine 
translation  required  (0),  or  human  translation  required  (-1).  M  can  be  augmented  by  multiplying 
additional  matrices  (layers)  such  as  normalized  bandwidth,  probability  of  connection  between 
system  pairs,  mission  capable  rate  for  systems,  normalized  cost,  system  reliability,  etc. 


Step  3 — Calculate  the  i-Score  /  =  X  X  my  ■ 

i=iy=i 

n  n  . 

Step  4 — Calculate  the  optimum  i-Score  lopt  =  X  X  my  where  Mopt 

i=Xj='  Mop, 


CiiSii 

J  J  l*«=max{*«} 


is  the 


maximally  upgraded  interoperability  matrix  (i.e.,  upgrade  all  spins  that  can  be  upgraded  in  light 
of  physical,  fiscal,  and  operational  constraints). 

Step  5 — Calculate  the  Interoperability  Gap  I  =  I  t  - 1 . 

Step  6 — Perform  interoperability  analysis  to  1)  determine  ways  of  closing  the  interoperability  gap 
through  spin  upgrades  or  using  common  systems,  2)  determine  average  interoperability  spin,  3) 


compare  operational  threads  through  a  normalized  i-Score,  or  4)  visualize  the  interoperability  of  a 
thread  by  graphing  it  on  an  Interoperability  Terrain  graph. 

While  i-Score  is  new,  and  has  not  yet  withstood  wide  academic  debate,  its  authors 
naturally  feel  it  has  promise  due  to  its  ability  1)  to  represent  different  types  of  interoperability,  2) 
to  provide  not  only  an  as-is  measurement  of  interoperability  but  also  a  realistic  theoretical 
optimum  interoperability  measurement,  3)  to  provide  a  mathematical  means  of  measuring 
interoperability  enabling  a  variety  of  optimization  techniques  to  be  used  to  improve  networks  of 
systems,  4)  to  provide  a  means  of  measuring  interoperability  for  homogeneous  and  heterogeneous 
networks  of  systems,  and  5)  to  provide  a  flexible  and  customizable  model  in  which  layers  of 
different  interoperability -related  parameters  can  be  analyzed. 

Interoperability  Measurement  Model  Summary 

In  order  to  more  easily  visualize  the  types,  strengths,  and  weaknesses  of  the  fourteen 
models  surveyed  in  this  paper  as  well  as  to  help  the  analyst  choose  the  model  appropriate  for 
his/her  application,  the  fourteen  models  described  above  are  summarized  in  two  tables  in  the 
appendix.  The  first  table  is  a  list  of  the  formats  of  the  interoperability  measures.  The  second 
table  is  a  more  comprehensive  table  which  more  fully  categorizes  the  interoperability  measures 
and  models.  The  categorization  of  the  models  in  the  second  table  is  based  upon  the  published, 
descriptive  statements  by  the  model’s  creator.  For  example,  if  the  author  of  a  model  did  not  state 
specifically  that  mathematical  operations  (e.g.,  optimization  techniques)  could  be  performed  on 
the  measurement  associated  with  the  model,  then  we  did  not  indicate  that  on  the  table  unless  it 
was  readily  apparent  (e.g.,  IAM). 

The  Mathematics  of  Interoperability  Measurement 

An  inspection  of  the  interoperability  measurement  model  summary  table  in  the  appendix 
indicates  that  most  authors  of  the  interoperability  measurement  models  were  not  focused  on 
applying  mathematical  methods  such  as  optimization  theory,  probability  theory,  or  complexity 
theory  to  their  model  or  having  an  analyst  apply  mathematics  to  their  specific  instance  of  the 
model.  Only  one  of  the  fourteen  models  ( i-Score )  was  designed  specifically  for  this  purpose, 
three  are  capable  of  having  some  mathematics  applied  (QoIM,  MCISI,  and  IAM),  and  only  seven 
have  a  numeric  or  partial  numeric  measurement  (SoIM,  QoIM,  MCISI,  IAM,  LCI,  LCIM,  NTI, 
and  i-Score).  We  believe  that  although  a  qualitative  framework  is  useful  as  a  starting  point,  the 
true  power  of  interoperability  measurement  is  in  post-measurement  analysis  founded  upon  strong 
mathematical  principles.  Many  large  networks  of  systems  are  already  in  existence,  but  a 
qualitative  approach  cannot  optimize  these  already  functioning  networks.  The  past  decade  has 
seen  numerous  improvements  in  linear  programming  techniques  which  can  be  applied  to  small  or 
extremely  large  networks.  Probability  theory  can  be  applied  to  our  networks  which  already 
interoperate  to  more  accurately  measure  their  true  interoperability  compensating  for  real-world 
degradations.  Finally,  techniques  related  to  complexity  theory  should  be  capitalized  upon  in 
order  to  measure  the  interoperability  of  either  self-emergent  networks,  self-synchronizing 
systems,  large  scale-free,  or  time-varying  networks. 

Institutionalization  of  Interoperability  Measurement  Models 

It  is  interesting  to  note  from  the  interoperability  measurement  summary  table  in  the 
appendix  that  only  two  models  (LISI  and  NMI)  appear  to  have  been  institutionalized  by  large 
organizations.  Even  then,  LISI  has  recently  been  de-institutionalized  within  the  Department  of 
Defense.  We  believe  there  are  many  explanations  for  this.  First,  many  interoperability 
measurements  that  are  being  performed  may  be  focused  on  system-to-system  measurements 


examining  individual  protocols,  applications,  or  physical  links.  This  type  of  measurement  falls 
into  the  category  of  interoperability  testing  and  is  done  at  a  very  low  level  on  the  spectrum  of 
interoperability  defined  by  some  of  the  models  reviewed  in  this  paper.  Nearly  all  of  the 
interoperability  measurement  models  reviewed  in  this  paper  are  more  suited  towards  higher-level 
characteristics  (e.g.,  social,  organizational,  or  procedural)  vice  lower-level  technical  ones. 

Second,  in  spite  of  the  interest  interoperability  has  received  over  the  past  three  decades, 
interoperability  measurement  appears  not  to  have  been  used  as  often  as  it  could  be.  Much 
attention  has  been  placed  upon  standards,  technical  reference  models,  and  common  protocols,  but 
we  still  have  networks  of  systems  which  are  using  those  accepted  standards  and  protocols  which 
are  not  fully  interoperable.  We’ve  measured  the  interoperability  of  two  systems,  but  have  not 
measured  their  interoperability  in  a  real-world  network.  We  also  have  not  measured  their  ability 
to  interoperate  in  self- formed,  emergent,  or  ad  hoc  networks.  Third,  it  may  be  that  such 
initiatives  as  the  Net-Ready  Key  Performance  Parameter  (NR-KPP)  and  Joint  Interoperability 
Test  Center  (JITC)  Certification  have  focused  attention  on  system-to-system  interoperability, 
adherence  to  standards,  and  interoperability  of  applications  within  an  as-is  network.  While  these 
initiatives  are  beneficial,  they  may  be  partially  deflecting  attention  and  money  away  from  some  of 
the  other  useful  types  of  measurements  described  by  the  models  summarized  in  this  paper  such  as 
operational  interoperability,  operational  interoperability  agility,  conceptual  model 
interoperability,  and  system-of-systems  interoperability.  Finally,  several  authors  stated  that  their 
models  were  in  their  infancy,  or  were  starting  points  for  further  research.  Indeed,  this  indicates 
that  further  refinement  of  the  current  set  of  models  is  required  before  widespread  use. 

Conclusions  and  Areas  for  Future  Work 

To  our  knowledge,  no  other  researcher  or  group  of  researcher  has  compiled  as  extensive 
of  a  summary  of  the  past  thirty  years  of  interoperability  measurement  research  as  can  be  found  in 
this  paper.  However,  many  very  well  written  papers,  reports,  and  notes  have  reviewed  and 
critiqued  sub-sets  of  the  fourteen  models  presented  in  this  summary  document  and  have  been 
cited  throughout  this  paper.  We  encourage  them  to  be  read  side-by-side  with  our  document.  Our 
study  of  all  the  materials  cited  in  this  research  paper  lead  us  to  the  following  conclusions  and 
recommendations  for  areas  which  merit  further  work. 

Refine  the  Field  of  Interoperability  Measurement  Research.  We  agree  with  Morris,  et  al.  in 
stating  that  there  is  a  need  for  the  community  to  work  together  to  compile  “a  complete  and 
consistent  set  of  interoperability  models.”  [Morris,  et  al.,  2004,  p.  47]  We  believe  it  might  be 
useful  to  form  a  working  group  (possibly  under  the  auspices  of  the  ICCRTS  or  INCOSE)  in 
which  interoperability  measurement  as  a  science  can  be  refined  and  promoted. 

Pursue  Mathematical  Methods  for  Interoperability  Measurement.  As  described  earlier  in 
this  paper,  the  mathematics  of  interoperability  measurement  have  largely  been  ignored  in  favor  of 
qualitative  measures  of  interoperability.  We  believe  that,  while  qualitative  measures  are  useful, 
quantitative  methods  such  as  linear  programming  and/or  non-linear  optimization,  probability 
theory,  graph  theory,  complexity  theory,  and  others  are  required  in  the  evolution  of 
interoperability  measurement.  The  measurement  of  interoperability  of  advanced  and  complex 
networks  of  systems  of  technology,  people,  and  organizations  is  a  critical  step  towards  the 
realization  of  truly  interoperable  systems. 

Develop  a  Method  of  Modeling  and  Measuring  Interoperability  of  Heterogeneous  Self- 
Forming  Networks.  An  abbreviated  search  indicates  that  more  and  more  researchers  are 
beginning  to  pay  attention  to  self- forming  networks.  Although  much  research  seems  to 
emphasize  homogeneous  networks,  a  framework  for  describing  and  a  method  for  measuring 


interoperability  of  heterogeneous  self-forming  networks  seems  to  be  a  useful  field  of  study.  The 
somewhat  recently  published  OIAM  could  be  an  ideal  non-technical  starting  point  as  it  measures 
the  ability  of  an  operational  network  of  forces  to  adapt  and  interoperate  with  other  forces. 
[Kingston,  Fewell,  and  Richer,  2005] 

Perform  Applied  Research  on  Extant  Models.  The  authors  of  the  models  described  in  this 
paper,  also  provided  one  or  two  case  studies,  military  operations,  or  other  examples  of  the 
application  of  their  model.  Many  of  those  applications  were  limited  in  scale  and  the  authors  of  at 
least  five  (QoIM,  LIST  OIM,  Stoplight,  and  OIAM)  of  the  models  stated  a  need  for  further 
application  of  their  model  in  order  to  validate  it  or  to  find  its  strengths  and  weaknesses.  [Mensh, 
Kite,  &  Darby,  1989,  p.  259],  [C4ISRAWG,  1998,  p.  8-1],  [Clark  &  Jones,  1999,  p.  11], 
[Hamilton,  Rosen,  &  summers,  2002,  p.  29],  [Kingston,  Fewell,  &  Richer,  2005,  p.  17]  For  this 
reason,  we  believe  that  more  research  showing  the  application  of  these  models  would  be  useful. 
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Appendix 


Interoperability  Definitions  (in  Chronological  Order) 


# 

Origin 

Definition 

Source 

1 

DoD 

The  ability  of  systems,  units,  or  forces  to  provide  services  to  and 
accept  services  from  other  systems,  units,  or  forces  and  to  use  the 
services  so  exchanged  to  enable  them  to  operate  effectively  together. 

(DoDD  2010.6,  1977), 

(DoDD  2010.6,  1980,  Enel.  2,  p.  2), 
(Amanowicz,  1996,  p.  280), 

(DoD  95), 

(DoD  98), 

(Leite,  1998,  p.  3), 

(Curts  1999,  p.  5.), 

(JV2020  ( JP 1  -02),  2000,  p.  15), 

(DoD  2001), 

(Clark  2001,  p.  2), 

(Fewell,  2003,  p.  2), 

(Kasunic,  2004,  p.  vii,  2,  32), 

(Morris,  et  al.,  2004,  p.  3,  7) 

2 

Other 

The  ability  of  one  system  to  receive  and  process  intelligible 
information  of  mutual  interest  transmitted  by  another  system. 

(Eldridge,  1978), 

(Kasunic  &  Anderson,  2004,  p.  32) 

3 

DoD 

Electronic  Interoperability.  A  special  form  of  interoperability  whereby 
two  or  more  electronic  equipments,  especially  communications 
equipments,  can  be  linked  together,  usually  through  common  interface 
characteristics  and  so  operate  the  one  to  the  other. 

(DoDD  2010.6,  1980,  Enel.  2,  p.  2) 

4 

DoD 

Logistic  Interoperability.  A  form  of  interoperability  whereby  the 
service  to  be  exchanged  is  assemblies,  components,  spares,  or  repair 
parts.  Logistic  interoperability  will  often  be  achieved  by  making  such 
assemblies,  components,  spares,  or  repair  parts  interchangeable,  but 
can  sometimes  be  a  capability  less  than  interchangeability  when  a 
degradation  of  performance  or  some  limitations  are  operationally 
acceptable. 

(DoDD  2010.6,  1980,  Enel.  2,  p.  3) 

5 

Other 

The  effort  required  to  couple  one  system  with  another. 

(McCall,  1980), 

(Kasunic  &  Anderson,  2004,  p.  32) 

6 

Other 

Interoperability  means  the  ability  of  two  or  more  parties,  machine  or 
human,  to  make  a  perfect  exchange  of  content.  Perfect  means  no 
perceptible  distortions  or  unintended  delays  between  content  origin, 
processing  and  use. 

(Poppel,  1987,  p.  1) 

7 

Standard 

The  ability  of  two  or  more  systems  or  components  to  exchange 
information  and  to  use  the  information  that  has  been  exchanged. 

(IEEE,  1990), 

(Kasunic  &  Anderson,  2004,  p.  32), 
(Kosanke,  2005,  p.  2) 

8 

Other 

Interoperability  among  components  of  large-scale,  distributed  systems 
is  the  ability  to  exchange  services  and  data  with  one  another. 

(Heiler,  1995,  p.  1) 

9 

Other 

The  ability  to  communicate  with  peer  systems  and  access  their 
functionality. 

(Vemadat,  1996), 

(Kosanke,  2005,  p.  2) 

10 

DoD 

Operational  Interoperability.  The  ability  of  systems,  units,  or  forces  to 
provide  services  to  or  access  services  from  other  systems,  units,  or 
forces,  and  to  use  the  services  to  operate  effectively  together. 

(DoD  96) 

11 

DoD 

Technical  Interoperability.  Interoperability  is  the  ability  of  systems  to 
provide  dynamic  interactive  information  and  data  exchange  among 

C4I  nodes  for  planning,  coordination,  integration,  and  execution  of 
Theater  Air  Missile  Defense  operations. 

(JTAMDO  97) 

12 

DoD 

Technical  Interoperability.  The  condition  achieved  among 
communications-electronics  systems  or  items  of  communications- 
electronics  equipment  when  information  or  services  can  be  exchanged 
directly  and  satisfactorily  between  them  and/or  their  users.  The 
degree  of  interoperability  should  be  defined  when  referring  to  specific 
cases. 

(DoD  98), 

(Leite,  1998,  p.  3), 

(DoD,  2001), 

(Morris,  et  al.,  2004,  p.  7-8), 

(JP  1-2,  2004,  p.  277) 

13 

DoD 

JCS  defines  interoperability  as  the  condition  achieved  between 
systems  when  information  or  services  are  exchanged  directly  and 
satisfactorily  between  the  systems  ad/or  their  users. 

(Curts  1999,  p.  9) 

14 

Standard 

The  ability  of  two  or  more  systems  of  elements  to  exchange 
information  and  to  use  the  information  that  has  been  exchanged. 

(IEEE  2000), 

(Morris,  et  al.,  2004,  p.  3) 

15 

Standard 

The  capability  for  units  of  equipment  to  work  together  to  do  useful 
functions. 

(IEEE,  2000), 

(Morris,  et  al.,  2004,  p.  3) 

16 

Standard 

The  capability,  promoted  but  not  guaranteed  by  joint  conformance 
with  a  given  set  of  standards,  that  enables  heterogeneous  equipment, 
generally  built  by  various  vendors,  to  work  together  in  a  network 
environment. 

(IEEE,  2000), 

(Morris  et  al.,  2004,  p.  7) 

17 

Standard 

The  ability  of  two  or  more  systems  or  components  to  exchange 
information  in  a  heterogeneous  network  and  use  that  information. 

(IEEE  2000), 

(Morris,  et  al.,  2004,  p.  7) 

18 

DoD 

The  ability  of  systems  to  work  together. 

(DSP,  2001,  p.  B4-7) 

19 

Standard 

The  ability  of  different  types  of  computers,  networks,  operating 
systems,  and  applications  to  work  together  effectively,  without  prior 
communication,  in  order  to  exchange  information  in  a  useful  and 
meaningful  manner.  There  are  three  aspects  of  interoperability: 
semantic,  structural,  and  syntactical. 

(Dublin  Core  Metadata  Glossary, 

2001) 

20 

DoD 

(1)  Ability  of  information  systems  to  communicate  with  each  other 
and  exchange  information.  (2)  Conditions,  achieved  in  varying  levels, 
when  information  systems  and/or  their  components  can  exchange 
information  directly  and  satisfactorily  among  them.  (3)  The  ability  to 
operate  software  and  exchange  information  in  a  heterogeneous 
network  (i.e.,  one  large  network  made  up  of  several  different  local 
area  networks).  (4)  Systems  or  programs  capable  of  exchanging 
information  and  operating  together  effectively. 

(GIG,  2001), 

(Morris,  et  al.,  2004,  p.  4) 

21 

DoD 

Programmatic  Interoperability.  Programmatic  interoperability 
encompasses  the  activities  related  to  the  management  of  one  program 
in  the  context  of  another  program. 

(Levine,  2003,  p.  4) 

22 

DoD 

Constructive  Interoperability.  Constructive  interoperability  addresses 
those  activities  related  to  construction  and  maintenance  of  one  system 
in  the  context  of  another  system.  Constructive  interoperability 
includes  the  common  use  of  architecture,  standards,  data 
specifications,  communication  protocols,  languages,  and  COTS 
products  to  build  interoperable  systems. 

(Levine,  2003,  p.  5) 

23 

DoD 

Operational  Interoperability.  Operational  Interoperability  refers  to  the 
activities  related  to  the  operation  of  a  system  in  the  context  of  other 
systems.  These  activities  include:  doctrine  governing  the  way  the 
system  is  used,  conventions  for  how  the  user  interprets  information 
derived  from  interoperating  systems  (i.e.,  the  semantics  of 
interoperation),  and  strategies  for  training  personnel  in  the  use  of 
interoperating  systems. 

(Levine,  2003,  p.  6) 

24 

DoD 

The  ability  of  systems  to  work  together. 

(Levine,  2003,  p.  26) 

25 

DoD 

The  ability  of  systems  to  exchange  and  use  services. 

(Levine,  2003,  p.  26) 

26 

DoD 

The  degree  to  which  a  set  of  communicating  systems  are  (i)  able  to 
exchange  specified  state  data,  and  (ii)  operate  on  that  state  data 
according  to  specified,  agreed  to,  operational  semantics. 

(Levine  2003,  p.  26) 

27 

Standard 

The  ability  to  integrate  data,  functionality  and  processes  with  respect 
to  their  semantics. 

(Berre,  et  al.,  2004,  p.  13) 

28 

DoD 

Ability  to  achieve  “cooperation”  is  generally  termed 
“interoperability.” 

(Carney,  2004,  Slide  #3) 

29 

DoD 

The  ability  of  one  services'  system  to  receive  and  process  intelligible 
information  of  mutual  interest  transmitted  by  another  service's  system. 

(Kasunic  &  Anderson,  2004,  p.  32) 

30 

DoD 

The  ability  of  a  set  of  communicating  entities  to  (1)  exchange 
specified  state  data  and  (2)  operate  on  that  state  data  according  to 
specified,  agreed-upon,  operational  semantics. 

(Morris,  et  al.,  2004,  p.  4) 

31 

Other 

Interoperability  is  defined  as  the  ability  of  two  or  more  systems  or 
components  to  exchange  information  and  to  use  the  information  that 
has  been  exchanged. 

(Blanc,  2005,  p.  2) 

32 

Other 

IDEAS  Project  defines  interoperability  as  the  ability  of  interaction 
between  enterprise  software  applications.  The  interoperability  is 
considered  achieved  if  interactions  can,  at  least,  take  place  at  three 
levels:  data,  application,  and  business  process  with  the  semantics 
defined  in  a  business  concept. 

(Blanc,  2005,  p.  2) 

33 

Other 

Ability  of  two  or  more  devices  to  work  together  in  one  or  more 
applications. 

(Kosanke,  2005,  p.  4) 

34 

DoD 

The  ability  to  operate  in  synergy  in  the  execution  of  assigned  tasks. 

(JP1-2,  2006,  p.  277) 

Interopera  bility  Types 


# 

Interoperability 

Type 

Source 

1 

Communications 

(LaVean,  1980,  p.  1448), 
(Kasunic  &  Anderson,  2004, 

P-  34) 

2 

Electronic 

(DoDD  2010.6,  1980,  Enel.  2, 

p.  2) 

3 

Logistics 

(DoDD  2010.6,  1980,  Enel.  2, 
p.  3) 

4 

Peacetime 

(LaVean,  1980,  p.  1450) 

5 

Systems 

(LaVean,  1980,  p.  1449), 
(Clothier,  1996,  1997), 

(Leite,  1998,  p.  1), 

(Curts,  1999,  p.  3), 

(GIG,  2001,  p.  32), 

(Clark,  2001,  p.  2), 

(Kasunic,  2004,  p.  1) 

6 

Telecommunications 

(LaVean,  1980,  p.  1449) 

7 

Multidatabase 

(Litwin  &  Abdellatif,  1986,  p. 
i) 

8 

Specification  Level 

(Wileden,  et  al.,  1989,  p.  1) 

9 

Object  Oriented 

(Konstantas,  1993,  p.  i) 

10 

High-Level 

(Konstantas,  1993,  p.  2) 

11 

Procedure  Oriented 

(Konstantas,  1993,  p.  4) 

12 

Semantic 

(Heiler,  1995,  p.  1) 

13 

Process 

(Clothier,  1996,  1997), 

(Clark,  2001,  p.  2) 

14 

System-to-System 

(Amanowicz,  1996,  p.  280), 
(Kasunic  &  Anderson,  2004, 
p.  17) 

15 

Information 

(Mathwick,  1997), 

(Curts,  1999,  p.  4), 

(DSP,  2001,  p.  B4-7) 

16 

Isolated 

(C4ISR,  1998), 

(Larsen,  2006,  p.  2) 

17 

Connected 

(C4ISR,  1998), 

(Larsen,  2006,  p.  2) 

18 

Functional 

(C4ISR,  1998), 

(GIG,  2001,  p.  22), 

(Clark,  2001,  p.  2), 

(Larsen,  2005,  p.  2) 

19 

Domain 

(C4ISR,  1998), 

(Larsen,  2006,  p.  2) 

20 

Enterprise 

(C4ISR,  1998), 

(Blanc,  2005,  p.  3), 

(Kosanke,  2005,  p.  8), 

(Larsen,  2006,  p.  2) 

21 

Data 

(ITSG,  1998), 

(Curts,  1999,  p.  4), 

(GIG,  2001,  p.  30), 

(Kasunic  &  Anderson,  2004, 
p.  4,  7,  34) 

22 

Joint 

(Leite,  1998,  p.  1), 

(GIG,  2001,  p.  49), 

(DSP,  2001,  p.  B4-18), 

(Kasunic  &  Anderson,  2004, 
p.  13-14) 

23 

Architecture 

(Curts,  1999,  p.  10) 

24 

Organizational 

(Clark,  1999,  p.  1), 

(Clark,  2001,  p.  1) 

25 

Technical 

(Clark,  1999,  p.  4), 

(GIG,  2001,  p.  22), 

(Clark,  2001,  p.  1), 

(Kinder,  2002,  p.  25), 

(Carney,  2004,  p.  16), 

(Kasunic  &  Anderson,  2004, 

p.  2) 

26 

Total 

(Curts,  1999,  p.  1) 

27 

Joint  Information 

(Nutwell,  2000) 

28 

Secure-Voice 

(GIG,  2001,  p.  33) 

29 

Non-GIG 

(GIG,  2001,  p.  29) 

30 

“Plug-and-Play” 

(GIG,  2001,  p.  47) 

31 

Coalition 

(GIG,  2001,  p.  48), 

(Fewell,  2003,  p.  1) 

32 

Information  Systems 

(DSP,  2001,  p.  B4-iii) 

33 

Materiel 

(DSP,  2001,  p-  B4-iii) 

34 

Doctrine 

(DSP,  2001,  p.  B4-iii) 

35 

Domain-Centered 

(DSP,  2001,  p.  B4-iii) 

36 

Mission-Centered 

(DSP,  2001,  p.  B4-iii) 

37 

International 

(DSP,  2001,  p.  B4-2) 

38 

Cultural 

(Clark,  2001,  p.  2) 

39 

Flexible 

(Clark,  2001,  p.  2) 

40 

Force 

(Clark,  2001,  p.  1) 

41 

Model 

(Clark,  2001,  p.  1) 

42 

Non-technological 

(Clark,  2001,  p.  1) 

43 

Planned 

(Clark,  2001,  p.  3) 

44 

Responsive 

(Clark,  2001,  p.2) 

45 

Cities 

(Kinder,  2002,  p.  18) 

46 

Horizontal 

(Kinder,  2002,  p.  27) 

47 

Intra-organisational 

(Kinder,  2002,  p.  23) 

48 

Public  Administration 

(Kinder,  2002,  p.  6) 

49 

Public  Service 

(Kinder,  2002,  p.  7) 

50 

Vertical 

(Kinder,  2002,  p.  27) 

51 

Constructive 

(Levine,  2003,  p.  5), 

(Carney,  2004,  p.  19), 

(Morris,  et  al.,  2004,  p.  1 1) 

52 

Operational 

(Levine,  2003,  p.  6), 

(Carney,  2004,  p.  19), 

(Kasunic  &  Anderson,  2004, 

P-  2), 

(Morris,  et  al.,  2004,  p.  1 1 ) 

53 

Transitive 

(Morris,  et  al.,  2004,  p.  28) 

54 

Programmatic 

(Levine,  2003,  p.  4), 

(Carney,  2004,  p.  19), 

(Morris,  et  al.,  2004,  p.  1 1 ) 

55 

System-of-Systems 

(Morris  et  al.,  2004,  p.  Cover) 

56 

Conceptual 

(Carney,  2004,  p.  18) 

57 

C4I 

(Kasunic  &  Anderson,  2004, 
p.  9) 

58 

Lower-layer 

(Kasunic  &  Anderson,  2004, 
p.  34) 

59 

Higher-layer 

(Kasunic  &  Anderson,  2004, 

P-  34) 

60 

Application 

(Kasunic  &  Anderson,  2004, 
p.  34), 

(Kosanke,  2005,  p.  4) 

61 

Product-to-Product 

(Kasunic  &  Anderson,  2004, 
p.  37) 

62 

Programmatic 

(Morris,  et  al.,  2004,  p.  33) 

63 

Constructive 

(Morris,  et  al.,  2004,  p.  35) 

64 

Coalition  C2 

(Larsen,  2006,  p.  1) 

Summary  of  Interoperability  Measure  Formats 


Method 

Acronym 

Date 

Measure 

Spectrum  of  Interoperability 

SoIM 

1980 

{1,2, 3, 4, 5, 6, 7}  per  system  pair 

Quantification  of  Interoperability 

QoIM 

1989 

x/y  ratio  for  each  of  7  components  where  x, 
y  are  positive  integers 

Mil  Comm.  &  Info  Systems  Interoperability 

MCISI 

1996 

Positive  integer  per  system  pair 

Levels  of  Information  System  Interoperability 

LISI 

1998 

Xny  per  info  system  where 

Xe  {General,  Expected,  Specific}, 

ne  {0,1,2, 3, 4}, 

ye{a...z} 

Interoperability  Assessment 

IAM 

1998 

Various  number  &  non-number  measures 
per  system  attribute 

Organisational  Interoperability 

OIM 

1999 

(0,1, 2, 3, 4}  per  organization 

NATO  Reference  Model  for  Interoperability 

NMI 

1999 

{0,1, 2, 3, 4}  per  info  system 

Stoplight 

Stoplight 

2002 

{Red,  Y ellow,  Orange,  Green}  per  legacy 
system 

Layers  of  Coalition  Interoperability 

LCI 

2003 

{1,2, 3, 4, 5, 6, 7, 8, 9}  per  coalition 

Levels  of  Conceptual  Interoperability 

LCIM 

2003 

{0,1, 2,3,4}  per  model 

System  of  Systems  Interoperability 

SoSI 

2004 

User  defined 

Non-technical  Interoperability 

NTI 

2004 

{1,2,4,8,12,16}  per  attribute  per  force  (for 
Terminology  and  ROE  attributes  only) 

Organisational  Interoperability  Agility 

OIA 

2005 

{0,1, 2, 3, 4}  per  organization 

Interoperability  Score 

i-Score 

2007 

Real  number  per  system,  operational  thread, 
network,  or  mission 

Summary  of  Interoperability  Measurement  Models 


SoIM 

QoIM 

MCISI 

LISI 

IAM 

OIM 

Stoplight 

LCI 

LCIM 

NMI 

SoSI 

NTI 

OIAM 

i-Score 

Year 

1980 

1989 

1996 

1998 

1998 

1999, 

2003 

2002 

2003 

2003 

2003 

2004 

2004 

2005 

2007 

Authors 

LaVean 

Mensh, 

Kite, 

Darby 

Amanowicz, 

Gajewski 

Various 

Leite 

Clark, 

Jones, 

Fewell 

Hamilton, 

Rosen, 

Summers 

Tolk 

Tolk, 

Muguira 

Various 

Morris,  et  al. 

Stewart,  et  al. 

Kingston, 

Fewell, 

Richer 

Ford,  et  al. 

Type  of  Interoperability  Measured 

Operational  Interoperability 

X 

Technical  Interoperability 

X 

X 

X 

X 

X 

X 

Operational  Interoperability  Agility 

X 

Non-Technical  Interoperability 

X 

Coalition  Interoperability 

X 

Conceptual  Interoperability 

X 

Programmatic  Interoperability 

X 

X 

Constructive  Interoperability 

X 

Multiple  Types 

X 

X 

X 

Type  of  Measurement(s) 

N/A 

Non-number 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Number 

X 

X 

X 

X 

X 

X 

X 

Aggregate  (roll  up  to  single  measure?) 

X 

X 

X 

Non-aggregate  or  multiple  measurement(s) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Basis  of  Measurement 

N/A 

System 

X 

X 

X 

X 

X 

System  Pair 

X 

X 

X 

X 

X 

X 

X 

More  than  two  systems 

X 

X 

X 

Mission,  scenario,  thread,  coalition,  unit,  etc. 

X 

X 

X 

X 

X 

X 

Math  Operations  Possible  on  Measurement(s)? 

Yes 

X 

No 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Maybe,  or  only  on  part  of  model 

X 

X 

X 

Institutionalization 

Government 

X 

X 

Industry 

Standards  Organization 

Other  or  unknown 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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Outline 
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■  Background 

■  Introduction 

■  Interoperability  Definitions 

■  Interoperability  Types 

■  Interoperability  Measurement  Centers  of  Gravity 

■  Interoperability  Measurement  Models 

■  Mathematics  of  Interoperability  Measurement 

■  Conclusion  &  Areas  for  Future  Work 
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Background 


U.S.  AIR  FORCE 


■  Interoperability  has  been  a  topic  of  concern  for  at  least  30  years 

■  GAO  has  told  Congress 

■  Interoperability  has  been  a  “longstanding  problem” 

■  “Services  historically  have  been  unable  to  communicate  effectively 
among  themselves  during  joint  operations” 

■  Interoperability  failures  in  “Korea,  the  Dominican  Republic, 
Vietnam. ..(and)  Grenada”,  “Persian  Gulf” 

■  DoD  continues  to  experience  interoperability  problems 

■  Much  work  has  already  been  done: 

■  Military  policy  created 

■  Interoperability  defined 

■  Types  of  interoperability  identified 

■  Frameworks,  methods,  models,  and  measures  for  interoperability 
created 
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Introduction 


U.S.  AIR  FORCE 


■  Our  paper  &  presentation  are  meant  to  be  reference  documents 

■  Interoperability  definitions,  types,  methods,  &  centers-of- 
gravity 

■  Only  highlights  of  the  methods  described  by  the  papers  surveyed 
are  presented  in  our  ICCRTS  paper  and  presentation 

■  Read  the  original  documents  for  a  full  understanding! 

■  Meticulous  bibliography  is  supplied  in  the  survey  paper 

■  We  focused  on  presenting  strengths  of  the  methods  surveyed 

■  Applicability  of  each  method  is  limited  to  its  authors’  intent 

■  No  one  method  will  be  useful  in  all  analytical  situations 


“Interoperability  will  never  be  an  analytically  useful  field  of  study 
until  it  is  defined  in  a  quantitative  way.”  (Presson,  1983) 
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Interoperability  Definitions 


U.S.  AIR  FORCE 


#  of  New  Interoperability  Definitions  Proposed  per  Year 

7 
6 
5 
4 
3 
2 
1 
0 


1977  1981 


1985  1989  1993  1997  2001  2005 


Surge  in  Interest 
in  Interoperability 
(1995  -  2003) 


Agreement  upon 
Definition  Reached? 
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Interoperability  Definitions 

(cont. . .) 


U.S.AIR  FORCE 


Most  popular  definition  of  interoperability 
(13  mentions)  of  34  definitions  identified  was... 


“The  ability  of  systems,  units,  or  forces  to  provide 
services  to  and  accept  services  from  other  systems, 
units,  or  forces  and  to  use  the  services  so  exchanged 
to  enable  them  to  operate  effectively  together.” 


Official  DoD  Definition  (JP1-02):  The  ability  to  operate  in 
synergy  in  the  execution  of  assigned  tasks . 


Integrity  -  Service  -  Excellence 


6 


Interoperability  Types 


W 


U.S.  AIR  FORCE 


Interoperability  Type 

Communications 

Electronic 

Logistics 

Peacetime 

Systems 

Telecommunications 

Multidatabase 

Specification  Level 

Object  Oriented 

High-Level 

Procedure  Oriented 

Semantic 

Process 

System-to-  System 

Information 

Isolated 

Connected 

Functional 

Domain 

Enterprise 

Data 

Coalition  C2 

Interoperability  Type 


Joint 


Architecture 


Cultural 


Total 


Joint  Information 


Secure- Voice 


Non-GIG 


“Plug-and-Play” 


Coalition 


Information  Systems 


Materiel 


Doctrine 


Domain-Centered 


Mission-Centered 


International 


C  Organizational^ 


Flexible 


Force 


Model 


Non-technological 


Interoperability  Type 


Planned 


Responsive 


Cities 


Horizontal 


Intra-organisational 


Public  Administration 


Public  Service 


Vertical 


Constructive 


C  ^Operational^.  > 


Transitive 


Programmatic 


System-of-Systems 


Conceptual 


C4I 


Lower-layer 


Higher-layer 


Application 


Product-to-Product 


Programmatic 


Constructive 


Common  Interoperability  Types 


Integrity  -  Service  -  Excellence 


7 


Interoperability  Centers  of  Gravity 


U.S.  AIR  FORCE 


Defense  Information  Systems  Agency 

Department  of  Defense 


Sil 

&  MANAGEMENT. 


Australian  Government 
Department  of  Defence 

Defence  Science  rind 
Technology  Organisation 


QinetiQ 


Air  Force  Institute  of  Technology 


=-  Software  Engineering  Institute  Carnegie  Mel  Ion 
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Interoperability  Measurement  Models 


U.S.  AIR  FORCE 


NOTE:  Unless  otherwise  noted,  all  quotes  and  figures  presented  with  each  model’s  summary 
on  the  successive  slides  are  attributed  to  the  author  of  model  being  summarized. 
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Interoperability  Measurement  Model  #1 


U.S.  AIR  FORCE 


SolM 

Interoperability  in  Defense  Communications,  IEEE  Trans.  Comm.,  1980 
Gilbert  E.  LaVean 

Type  of  Interoperability:  Communication  System 

Motivation:  Addresses  interoperability  between  communications  systems 
supporting  U.S.  DoD,  U.S.  Civil  Government,  and  Allies. 

Abstract:  “A  spectrum  of  interoperability  is  presented  that  permits  system 
objectives  to  be  stated  in  more  precise  terms  and  also  provides  a  basis  for  cost 
versus  benefit  analysis.” 

Contribution:  Possibly  first  to  recognize  and  describe  levels  of  interoperability — 
1)  separate  systems,  2)  shared  resources,  3)  gateways,  4)  multiple  entry  points, 
5)  conformal/compatible  systems,  6)  completely  interoperable  systems,  and  7) 
same  system. 

DISf 
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Interoperability  Measurement  Model  #2 


U.S.  AIR  FORCE 


QolM 

The  Quantification  of  Interoperability,  Naval  Engineers  Journal ,  1989 
Dennis  R.  Mensh,  Robert  S.  Kite,  &  Paul  H.  Darby 

Type  of  Interoperability:  Battle  Force  C3  Systems,  Units,  &  Forces 
Motivation:  Joint  Pub  1 

Abstract:  Provides  “an  analysis  tool  to  enable  specific  detailed  analysis  of  the 
interoperability  of  BFC3  systems,  units,  or  forces  for  the  purpose  of  uncovering 
and  resolving  interoperability  issues  and  problems  in  the  U.S.  Navy,  Joint,  and 
Allied  arenas.” 


Contribution:  Possibly  first  to  define  components  of  interoperability  to  capture 
the  “totality”  of  interoperability;  attached  MOEs  in  the  form  of  logic  equations  to 
each  component  in  order  to  perform  interoperability  analysis  of  Battle  Force  C3 
systems  using  actual  exercise  data. 
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QolM 


U.S.  AIR  FORCE 
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Interoperability  Measurement  Model  #3 


U.S.  AIR  FORCE 


MCISI 

Military  Communications  &  Information  Systems  Interoperability,  MILCOM,  1996 
Col.  Marek  Manaowicz  &  Col.  Piotr  Gajewski 

Type  of  Interoperability:  Military  Communication  &  Information  Systems  (CIS) 

Motivation:  Military  CIS  interoperability  between  national,  multinational,  and 
allied  forces  are  required  in  support  of  traditional  military  C2  as  well  as  in  support 
of  international  dialogue  &  cooperation,  humanitarian  aid,  and  peacekeeping. 

Abstract:  General  concept  of  military  CIS  interoperability  modeling  is  presented. 

Contribution:  Possibly  first  to  recognize  that  taxonomic  methods  are  useful  and 
that  “great  amounts”  of  data  are  needed  to  perform  proper  interoperability 
analysis.  Also,  possibly  first  to  express  an  interoperability  measure  as  the 
distance  between  systems,  measured  in  terms  of  their  “features.”  Showed  that 
dendrites  can  be  used  to  pictorially  describe  the  relationships  of  CIS  systems. 
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MCISI 
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Interoperability  Measurement  Model  #4 


U.S.  AIR  FORCE 


Levels  of  Information  Systems  Interoperability,  C4ISR  AWG,  1998 
C4ISR  Architecture  Working  Group  co-chaired  by  J6  and  ASD(C3I)/CISA 

Type  of  Interoperability:  Information  Systems  (IS) 


Motivation:  Joint  Vision  2010  -  “speed  and  accuracy  of  prioritizing  and 
transferring  data  brought  about  by  advances  in  technology.” 

Abstract:  A  “practical  assessment  process  for  determining  the  interoperability 
maturity  level  or  ‘metric’  of  a  given  system  or  system  pair,  and  we  lack  a  means 
for  the  community  to  work  collaboratively  toward  achieving  higher  states  of 
assured  Joint  interoperability.” 


Contribution:  Possibly  first  complete  reference  model  for  IS  interoperability. 
Defined  5  levels  of  interoperability  measured  across  4  attributes.  Described 
succinct  metrics  conveying  interoperability  of  a  single  IS  or  system  pair.  LISI 
institutionalized  by  DoD  in  support  of  JCIDS.  Complements  DoDAF. 
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LISI 


U.S.  AIR  FORCE 
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Interoperability  Measurement  Model  #5 


U.S.  AIR  FORCE 


1AM 

Interoperability  Assessment,  Proc.  66th  MORS,  1998  (revised  Aug  2003) 

Michael  J.  Leite 

Type  of  Interoperability:  System 

Motivation:  CINCLANTFLT  stated  in  1998  that  “there  is  no  focus  on  battle  group 
and  Joint  interoperability.” 

Abstract:  “A  methodology  that  characterizes  system  interoperability  deficiencies 
through  the  measurement  and  quantification  of  a  set  of  interoperability  system 
components.” 

Contribution:  Defined  7  degrees  of  interoperability  and  9  components  of 
interoperability.  Provided  some  equations  which  relate  to  the  9  components. 
Possibly  the  most  valuable  contribution  was  a  flowcharted  interoperability 
assessment  process  whose  utility  was  demonstrated  by  a  brief  application  to  a 
Navy  Theater  Ballistic  Missile  Defense  case  study. 
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1AM 


U.S.  AIR  FORCE 
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Interoperability  Measurement  Model  #6 


U.S.  AIR  FORCE 


OIM 

Organisational  Interoperability  Maturity  Model  for  C2,  Proc.  3rd  ICCRTS,  1999 
Thea  Clark  &  Richard  Jones 

Type  of  Interoperability:  Organizational 

Motivation:  “The  primary  challenge  of  conducting  joint  operations  is  increasingly 
summed  up  in  one  word,  interoperability.”  While  earlier  work  focused  on  system 
and  technical  interoperability,  this  paper  addresses  “issues  associated  with 
interoperability  at  the  organisational  level.” 

Abstract:  “Understanding  organisational  interoperability  is... vital  for  the  effective 
command  and  control  of... task  forces.” 


Contribution:  OIM  extends  LISI  to  cover  organizational  interoperability.  Like 
LISI,  defines  5  levels  and  4  attributes.  OIM  updated  in  2003  by  Suzanne  Fewell 
and  Thea  Clark  and  re-published  in  Proc.  9th  ICCRTS. 
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OIM 


U.S.  AIR  FORCE 
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Interoperability  Measurement  Model  #7 


U.S.  AIR  FORCE 


NMI 

NATO  C3  Technical  Architecture  Reference  Model  for  Interoperability,  1999,  2003 
NATO  Consultation,  Command,  and  Control  Agency  (NC3A) 

Type  of  Interoperability:  Technical,  Data 


Motivation:  “To  establish  measures  of  merit  to  evaluate  the  degree  of 
interoperability  between  two  existing  systems  by  applying  standard  means.” 
(Tolk  &  Muguira) 

Abstract:  “Enhancing  operational  effectiveness  by  structuring  and  automating 
the  exchange  and  interpretation  of  data.”  (Morris,  et  al.) 


Contribution:  The  original  NMI  complements  LISI  by  describing  5  levels  of  data 
interoperability.  NMI  provides  categories  of  elementary  services  which  form  a 
basis  of  interoperability  profiles.  According  to  Morris,  et  al.,  NMI  was  updated  in 
2003  to  “closely  reflect  the  LISI  model.”  NMI  is  no  longer  available  on-line. 
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Interoperability  Measurement  Model  #8 


U.S.  AIR  FORCE 


Stoplight 

An  Interoperability  Roadmap  for  C4ISR  Legacy  Systems,  Acq.  Rev.  Qtrly.,  2002 
John  Hamilton,  Jerome  Rosen,  &  Paul  Summers 

Type  of  Interoperability:  C4ISR  Legacy  System 

Motivation:  “The  DoD  has  made  tremendous  interoperability  gains... without  a 
way  to  assess  the  status  of  interoperability.. it  is  difficult  to  quantify  this 
progress... interoperability  successes  are  easily  overlooked.” 

Abstract:  “Most  systems  developed  today  meet  the  interoperability  requirements 
...specified  in  their  ORDs...a  set  of  metrics... would  highlight  the  successes  of 
the  many  agencies  that  have  labored  to  produce  interoperable  systems.” 

Contribution:  A  simple  model  which  assigns  a  single  color  code  to  a  system 
which  indicates  how  well  that  system  meets  operational  and  acquisition 
interoperability  requirements.  Possibly  first  model  to  directly  address  the 
relationship  between  requirements  and  interoperability. 
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Stoplight 


U.S.  AIR  FORCE 
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Interoperability  Measurement  Model  #9 


U.S.  AIR  FORCE 


LCI 

Beyond  Technical  Interoperability — Introducing  a  Reference  Model  for 
Measures  of  Merit  for  Coalition  Interoperability,  Proc.  8th  ICCRTS,  2003 
Andreas  Tolk 

Type  of  Interoperability:  Coalition 

Motivation:  “Interoperability  is  definitely  not  limited  to  the  technical  domain,  but 
is  dependent  on  organizational  aspects  as  well.” 

Abstract:  “A  framework  to  deal  with  possible  measures  of  merit  to  be  used  to 
deal  with  the  various  layers  of  semantic  interoperability  in  coalition  operations.” 

Contribution:  Emphasizes  the  importance  of  measures  of  merit  in  measuring 
interoperability.  Defines  a  framework  of  9  levels  for  measuring  interoperability 
which  shows  the  relationship  between  organizational  and  technical 


interoperability.  Describes  the  continuum  between  technical  and  organizational 
interoperability. 
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LCI 


U.S.  AIR  FORCE 
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©  2002  VMASC 
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Interoperability  Measurement  Model  #10 


U.S.  AIR  FORCE 


LCIM 

The  Levels  of  Conceptual  Interoperability  Model,  Proc.  2003  Fall  SIW,  2003 
Andreas  Tolk  &  James  Muguira 

Type  of  Interoperability:  Conceptual 

Motivation:  “In  order  to  achieve  meaningful  interoperability  of  simulation 
systems  on  the  technical  level,  composability  of  the  underlying  conceptual 
models  is  a  necessary  requirement.” 

Abstract:  “...a  general  model  dealing  with  various  levels  of  conceptual 
interoperability  that  goes  beyond  the  technical  reference  models.” 

Contribution:  Possibly  the  first  model  to  “bridge  the  gap  between 
implementation  focused  methods  and  conceptual  models.”  A  layered  model 
which  enhances  the  DoD  Net-Centric  Data  Strategy  for  the  Global  Information 
Grid. 
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LCIM 


U.S.  AIR  FORCE 
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Interoperability  Measurement  Model  #1 1 


U.S.  AIR  FORCE 


SoSI 

System  of  Systems  (SoSI):  Final  Report,  CMU  Tech.  Report,  2004 
Edwin  Morris,  Linda  Levine,  Craig  Meyers,  Pat  Place,  &  Dan  Plakosh 

Type  of  Interoperability:  Technical,  Operational,  Constructive,  &  Programmatic 

Motivation:  “Interoperability  must  occur  at  multiple  levels  within  and  across 
programs,  and  not  solely  in  the  context  of  a  system  construction.” 

Abstract:  “In  order  to  have  interoperability  between  operational  systems,  one 
must  introduce — and  address — the  full  scope  of  interoperability  between  those 
organizations  that  participate  in  the  acquisition  of  systems.” 

Contribution:  Proposed  that  a  set  of  models  is  needed  to  “collectively  address 
all  of  the  dimensions  of  interoperability.”  Introduced  the  concept  of  an 
interoperability  backplane  which  includes  environmental  factors  such  as  policy 
and  standards,  especially  those  prescribed  by  the  program  executive  officer. 


-=  Software  Engineering  Institute 


Carnegie  Mel  Ion 
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SoSI 


U.S.  AIR  FORCE 


Activities  performed  to  manage  the  acquisi¬ 
tion  of  a  system.  Focus  is  on  contracts, 
incentives,  and  practices  such  as  risk 
management. 
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Interoperability  Measurement  Model  #12 


U.S.  AIR  FORCE 


NTI 


Non-technical  Interoperability  in  Multinational  Forces,  Proc.  9th  ICCRTS,  2004 
K.  Stewart,  H.  Clarke,  P.  Goillau,  N.  Varrall,  and  M.  Widdowson 

Type  of  Interoperability:  Non-technical 

Motivation:  “Interoperability  in  multinational  forces  generally  refers  to 
compatibility  of  hardware  and  software.  Connectivity  alone,  however,  does  not 
confer  capability  and  must  be  accompanied  by  interoperability  of  people, 
process,  and  organisation.” 

Abstract:  “A  valid  framework  describing  factors  that  underpin  NTI.” 

Contribution:  Extends  OIM  by  describing  attributes  or  factors  of  non-technical 
interoperability,  pertaining  to  multinational  forces,  largely  gleaned  from  interviews 
with  45  British  officers  ranging  in  rank  from  Army  Captain  to  three-star  general. 
All  had  served  in  multinational  operations/settings.  _ 


[dstl] 
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A/77 


U.S.  AIR  FORCE 
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Interoperability  Measurement  Model  #13 


U.S.  AIR  FORCE 


01AM 

An  Organisational  Interoperability  Agility  Model,  Proc.  10th  ICCRTS,  2004 
Gina  Kingston,  Suzanne  Fewell,  &  Warren  Richer 

Type  of  Interoperability:  Organizational 


Motivation:  “Joint,  combined  and  coalition  operations  are  now  the  rule. 
Coalitions  are  often  formed  on  an  ad  hoc  basis... with  partners  joining  and 
leaving  or  scaling  their  commitments  during  the  course  of  the... operation.”  “This 
requires  the  development... of  modelling  and  measuring  techniques  to  assess 
the  impact.” 


Abstract:  A  model  that  “aims  to  capture  the  dynamic  aspects  of  working  in 
coalitions  including  the  ability  of  an  organisation  to  contribute  to  the  rapid 
formation  and  reformation  of  coalitions,  including  novel  ones.” 


Contribution:  Possibly  first  to  define  a  framework  for  measuring  how  capable, 
or  agile,  an  organization  is  with  respect  to  joining  a  coalition. 
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OIAM 


U.S.  AIR  FORCE 


Level 

Preparation  (indudes 
Understanding) 

Command  and 
Coordination 

Ethos 

4  Dynamic 

Compatible  doctrine  developed 
with  at  least  one  organisation 
across  each  organisational  and 
cultural  grouping  and  the  ability 
to  rapidly  achieve  this  with  most 
of  the  remaining  organisations 
for  each  activity  relevant  to  the 
context,  and  to  adjust  doctrine 
dunng  operations 

Wide  variety  of  relevant 
collective  experience  working 
with  other  organisations  across 
most  organisational  and  cultural 
groups  within  a  vanety  of 
contexts  including  for  different 
operational  types  and  cross 
organisational  bodies  relevant 
to  the  context 

High  awareness  of  future  01 
requirements  OIA  ANe  to 
rapidly  plan  and  train  to  achieve 
required  01  levels. 

Mechanisms  to  work  together 
and  share  information  already 
m  place  and  practiced  with  at 
least  one  organisation  across 
each  organisational  and 
social'cultural  grouping  and  the 
ability  to  rapidly  develop  them 
with  other  organisations  across 
most  organisational  groups. 

Mechanisms  for 
interacting  with 
management/C2 
styles  with  at  least  one 
organisation  in  each 
organisational  group 
and  the  ability  to 
rapidly  achieve  this 
with  most  other 
organisations 2  within 
the  context 

Practice  at  timely 
changing  C2  styles. 
Ability  to  exploit 
different  C2  styles  in 
different  parts  of  the 
organisation  and  to 
adjust  and  exploit  C2 
structures,  styles  and 
staffing  dunng 
operations. 

Ability  to  staff  C2 
structures  from  the 
whole  organisation 

Writing  to  operate  with 
any  organisation  in  the 
context  (including 
partners  with  which  no 
formal  doctrine  has 
been  developed).  This 
includes  a  willingness 
to  share  required 
information  with 
relevant  individuals 
within  most 
organisational  groups 
and  to  share 
information  across 
most  information 
classes  in  recognition 
that  a  common 
purpose  is  being 
served. 

Willing  to 
accommodate 
organisational 
differences  and  with 
expenence  negotiating 
with  different 
organisations. 

Accustomed  at 
adapting  fo  changes  in 
doctrine 

Willing  to 
accommodate 
differences  in  goals 
and  values  and  with 
techniques  for 
recognising  a  common 
purpose  while  working 
with  organisations  with 
multiple  conflicting 
goals. 

Level 

Preparation  (indudes 
Understanding) 

Command  and 
Coordination 

Ethos 

3  Open 

Compatible 

doctrine/procedures  developed 
with  at  least  one  organisation 
from  most  organisational  and 
cultural  groups  i.e.  the 
organisation  has  multiple  sets 
of  doctnne/procedures  covenng 
many  activities  relevant  to  the 
context. 

Aware  of  future  01 
requirements  and  have 
measures  and  metrics  to 
support  OIA. 

Have  mechanisms  to  develop 
new  workable 

doctnne/procedures  with  other 
organisations.  Ability  to  develop 
doctnne  for  new  activities  in 
advance  of  operations. 

Considerable  vanety  of  relevant 
collective  experience  working 
with  other  organisations  across 
most  organisational  and  cultural 
groups. 

Mechanisms  for 
interacting  with 
management/C2 
styles  with  at  least  one 
organisation  from 
most  organisational 
groups. 

Familianty  with 
different  C2  styles  and 
structures.  Ability  to 
adjust  C2  structures 
during  operations. 

Writing  to  operate  with 
most  organisations 
relevant  to  the  context 
across  all  of  the 
oryanrsafiona/  groups 
and  to  share  most 
classes  of  information 
with  them. 

Willing  to  adapt  to 
changes  in  doctrine 
and  procedures. 
Practiced  at  gaining 
new  skills  Encouraged 
to  generate  alternative 
mechanisms  for 
interactions 

Willing  to  recognising 
that  a  common 
purpose  may  exist 
while  accommodating 
differences  in  goals 
and  values  and 
working  with 
organisations  whose 
goals  conflict  with  each 
other. 

2 

Accommodating 

Preparation  for  future  01  is 
broadly  based,  but  awareness 
of  future  01  requirements  is  still 
limited 

For  most  activities  relevant  to 
the  context,  compatible 
doctnne/procedures  have  been 
developed  with  at  least  one 
organisation  from  most 
organisational  and  cultural 
groups  i.e.  multiple  sets  of 
doctnne/procedures  have  been 
developed 

A  variety  of  relevant  collective 
expenence  working  with  other 
organisations  across 
organisational  and  cultural 
groups 

Mechanisms  for 
interacting  with  the 
management/C2 
styles  of  organisations 
from  across  some 
organisational  groups. 

Willing  to  adapt  to 
different  C2  styles 

Able  to  adjust  C2 
structures  before 
operations  and  C2 
staffing  dunng 
operations 

Willing  to  operate  with 
at  least  one 
organisation  from  most 
organisational  groups 
and  to  share  some 
classes  of  information 
with  them  Encouraged 
to  develop  new  skills 

Willing  to  develop  new 
methods  and 
procedures  for 
interacting  with  the 
other  organisations. 

Level 

Preparation  (indudes 
Understanding) 

Command  and 
Coordination 

Ethos 

1  Amenable 

Preparation  for  future  01  is  ad- 
hoc.  with  limited  ability  to 
gather  information  about  future 
requirements. 

For  some  activities  relevant  to 
the  context,  there  is  more  than 
one  partner  organisation  with 
which  at  least  one  set  of 
compatible  doctnne/procedures 
has  been  developed 

Limited  collective  expenence 
working  with  organisations  from 
other  organisational  and 
cultural  groups. 

Acceptance  of  more 
than  one  command 
and  co-ordination 
style 

Mechanisms  for 
interacting  with  more 
than  one 

management/C2  style. 

Willing  to  operate  with 
a  limited  number  of 
alternative  partners 
e  g.  partners  without 
doctnne  and  to  share 
limited  information. 

Writing  to  adapt  to 
minor  changes  in 
doctrine  and 
procedures. 

Only  willing  to 
accommodate  minor 
differences  in  goals 
and  values. 

0  Static 

Almost  no  awareness  of.  or 
planning  for,  future  01 
requirements. 

For  each  activity  relevant  to  the 
context,  there  is  at  most  one 
partner  organisation  with  which 
a  set  of  compatible  doctnne  has 
been  developed 

Almost  no  collective  expenence 
working  with  any  other 
organisational  or  cultural  group 

Only  one  command 
and  co-ordination  style 
predominantly  used 
and  widely  accepted 

Unwilling  to 
accommodate 
different  C2  styles 

Not  willing  lo  operate 
with  alternative 
partners  e  g.  partners 
with  which  no  formal 
doctnne  has  been 
developed  Unwilling  to 
change  organisational 
structure  or  practices 
to  accommodate 
others  or  share 
information. 

Generally  unwilling  to 
change  goals  and 
allegiances  and 
accommodate  different 
.  j  im 
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Interoperability  Measurement  Model  #14 


U.S.  AIR  FORCE 


i-Score 

The  Interoperability  Score,  Proc.  2007  CSER,  2007 
Thomas  Ford,  John  Colombi,  Scott  Graham,  &  David  Jacques 

Type  of  Interoperability:  Technological,  Biological,  Organizational,  & 
Environmental 

Motivation:  In  performing  network-centric  operations,  “clearly  an  important 
factor  is  improving  interoperability  of  systems  of  all  types.”  Extant  interoperability 
measurement  methods  are  “more  qualitative  than  quantitative.” 


Abstract:  “A  generalized  measure  of  the  interoperability  of  systems  of  all  types, 
supporting  an  operational  thread.” 


Contribution:  Possibly  the  first  to  define  a  strictly  quantitative  means  of 
measuring  the  interoperability  of  a  heterogeneous  network  of  systems  in  the 
context  of  the  operation  those  systems  support.  Additionally,  proposes  a  means 
of  defining  the  maximum  measure  of  interoperability  in  light  of  mrri 

operational  and  technical  constraints. 


Atn  FOMDC  INSTITUTE  Of  KSHNOLOBf 
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U.S.  AIR  FORCE 


Interoperability  Measurement  Summaries 


Method 

Measure 

Spectrum  of  Interop.,  1980 

{1,2, 3, 4, 5, 6, 7}  per  system  pair 

Quantification  of  Interop.,  1989 

x/y  ratio  per  component 

Mil  Comm.  &  Info  Systems  Interop.,  1996 

Pos.  integer  per  system  pair 

Levels  of  Info.  System  Interop.,  1998 

{G,  E,  S}i3>{0...4}#{a...z}  per  info  system 

Interoperability  Assessment,  1998 

Various  number  &  non-number  measures 

Organisational  Interop.,  1999,  2003 

(0,1, 2, 3, 4}  per  organization 

NATO  Ref.  Model  for  Interop.,  1999,  2003 

{0,1, 2, 3, 4}  per  info  system 

Stoplight,  2002 

{R,  Y,  O,  G}  per  legacy  system 

Layers  of  Coalition  Interop.,  2003 

{1,2, 3, 4, 5, 6, 7, 8, 9}  per  coalition 

Levels  of  Conceptual  Interop.,  2003 

{0,1, 2, 3, 4}  per  model 

System  of  Systems  Interop.,  2004 

User  defined 

Non-technical  Interop.,  2004 

{1,  2,  4,  8, 12,  16}  per  attribute  per  force 

Organisational  Interop.  Agility,  2005 

{0,1, 2, 3, 4}  per  organization 

i-Score,  2007 

Real  number  per  operational  thread 
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Mathematics  of  Interoperability  Measurement 


U.S.  AIR  FORCE 


■  Graph  Theory 

■  Optimization  Theory 

■  Probability  Theory 

■  Matrix  Methods 

■  Mathematical  Logic 

■  Complexity  Theory 

■  Metrology 

■  Measures  of  Merit 
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Conclusion/ Areas  for  Future  Work 


U.S.  AIR  FORCE 


i.  Refine  the  field  of  interoperability  measurement 

■  “a  complete  and  consistent  set  of  interoperability  models”  is 
needed.  (Morris  et  al,  2004) 


2.  Pursue  mathematical  methods  for  interoperability  measurement 

■  Qualitative  measures  are  useful,  but  quantitative  methods  are 
required  as  the  next  step  in  interoperability  measurement 


3.  A  method  of  measure  interoperability  of  self-forming  networks  is 
needed 

■  OIAM  is  a  starting  point 


4.  Perform  applied  research  on  extant  models 

■  Necessary  to  flesh  out  the  strengths  and  weaknesses  of  the  models 
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Questions? 


I  welcome  comments  &  criticisms! 
thomas.ford@afit.edu 


